Unfair rejections on MTurk

A new study examines the relationship between work rejection and (mis) trust on MTurk. The authors find seven main risk factors that lead workers to feel any rejections are unfair. For Requesters, this list should serve as a ‘checklist’ for making sure that your HIT is “Turker Ready”. Here are the seven:

1. Flaws in task or interface design

2. Unclear evaluation criteria

3. Unresponsive, arbitrary resolution of rejections (lack of response from Requesters when contacted by a worker)

4. Lack of information on Requesters
5. Inexperienced and Unfamiliar Requesters (fear of ‘old wine in new bottles’ or existing bad Requesters creating new accounts)
6. Tasks with poor return (interestingly, longer tasks are seen as having ‘poor return’ as a rejection after a Worker  puts in a lot of time to complete the study is a huge loss of income)
7. Prioritizing efficiency over quality (Workers want to complete tasks efficiency, Requesters worry–generally needlessly–that this leads to reduced quality).

Lots of good solutions presented, many of which require cooperation by Amazon.

McInnis, Brian, Dan Cosley, Chaebong Nam, and Gilly Leshed. “Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers’ Experiences in Amazon Mechanical Turk.” In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2271-2282. ACM, 2016.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s