A paper to be presented at ACM in Atlanta and available for download here has some great tips for using MTurk and ensuring results are valid. The study itself is quite interesting, but the task was complex. Workers needed to learn how to use an interface to find out specific information, and once they learned it they were given 15 minutes to use the interface and find information. There was quite a bit of work that was part of this HIT.
Specific tips, particularly for a longer study:
-the researchers required participants had completed at least 50 hits
-participants were paid an average of $4 with a $1 bonus paid to people who completed the study
-the researchers used numerous satisficing checks (19!)
-the researchers providing a brief training phase–more than 500 people started the training, and just about 200 completed the actual HIT.
As we know from previous posts, $4 (or $5 with a bonus) for 15-16 minutes work is quite acceptable, so bravo to these Computer Science researchers from UCSB, U of Illinois and SA technolgoies.