Requester Reputation

I’ve been reading a lot about Worker Reputation (for example, see Peer, Vosgerau and Acquisti, abstract here). There are good reasons to use Workers with completion rates above 95% and total HITs above 500, such as better attention (as seen by successful completion of novel Attention Chekcs) and a greater commitment to doing good research on MTurk than others.

Most Workers know that there exists a place to know more about Requesters, and that place is the Turkopticon . Think of it as a Yelp for Workers to rate Requesters on three areas:

1. communicativity: How responsive has this requester been to communications or concerns a Worker has raised?
2. generosity: How well has this requester paid for the amount of time their HITs take?
3. fairness: How fair has this requester been in approving or rejecting work?
4. promptness: How promptly has this requester approved and paid for work?

The results aren’t cumulative, but instead numerous reports for an individual Requester can be found.

Requesters: this is a valuable source for you. You can see that Workers are OK with your work (and if they are, then more Workers will likely want to do your HITS…but we need to study that..). You can see areas where you can improve. For example, my own reviews showed that I did not pay well for early studies. As I’ve done more work on MTurk, I’ve changed that.

Here is a snippet of my reviews:
Screen Shot 2015-02-24 at 8.49.21 AM

On one of these studies, I had put my code in the center of the study as one of the attention checks and as you can see one of the people who took my study didn’t see it. This person passed another attention check so they got paid. Plus, I learned once I did several HITs as a Worker that putting a code in the center of a survey is just not a smart way to do it. So I changed my method after that.

Final note: you do need to sign up, but only email information is necessary.

One thought on “Requester Reputation

  1. “The results aren’t cumulative, but instead numerous reports for an individual Requester can be found.”

    There are cumulative average scores displayed in workers’ tools (and via the ‘Averages’ links visible in your screenshot) in addition to the individual reviews.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s