I’ve been reading a lot about Worker Reputation (for example, see Peer, Vosgerau and Acquisti, abstract here). There are good reasons to use Workers with completion rates above 95% and total HITs above 500, such as better attention (as seen by successful completion of novel Attention Chekcs) and a greater commitment to doing good research on MTurk than others.
Most Workers know that there exists a place to know more about Requesters, and that place is the Turkopticon . Think of it as a Yelp for Workers to rate Requesters on three areas:
1. communicativity: How responsive has this requester been to communications or concerns a Worker has raised?
2. generosity: How well has this requester paid for the amount of time their HITs take?
3. fairness: How fair has this requester been in approving or rejecting work?
4. promptness: How promptly has this requester approved and paid for work?
The results aren’t cumulative, but instead numerous reports for an individual Requester can be found.
Requesters: this is a valuable source for you. You can see that Workers are OK with your work (and if they are, then more Workers will likely want to do your HITS…but we need to study that..). You can see areas where you can improve. For example, my own reviews showed that I did not pay well for early studies. As I’ve done more work on MTurk, I’ve changed that.
On one of these studies, I had put my code in the center of the study as one of the attention checks and as you can see one of the people who took my study didn’t see it. This person passed another attention check so they got paid. Plus, I learned once I did several HITs as a Worker that putting a code in the center of a survey is just not a smart way to do it. So I changed my method after that.
Final note: you do need to sign up, but only email information is necessary.