So the very respected Pew Center put together a report about MTurk. Certainly a credible organization like Pew is a great source for researchers to legitimize the use of the tool. There are some great insights and some head scratchers in the study.
- Insight:” top five most active requesters, all businesses, accounted for more than half of all the HIT group postings on the site “
- Head scratcher: from the first page: “During the week of our analysis of content on the site, academics were responsible for slightly more of the HIT groups posted (36%) than businesses (31%). ” This does not reflect what I find–I find about 10% of HITS are for academic studies. The explanation of how they figured out what is an academic study indicates that they had 95% intercoder reliability. It still does not jive with my experience.
- Insight: that statistic that there are only 10,000 (or 40,000) active Turkers is not supported in this study: “According to Alexa.com – a web tracking company also owned by Amazon – Mechanical Turk had about 750,000 unique visitors worldwide in December 2015 alone. Of course not all of those visitors participated in the marketplace. Some may have visited to see how the site works. However, the Alexa data give at least circumstantial evidence that many were there to earn money. The average user visited the site almost eight times each in December and spent more than 33 minutes per day there.”
- Head scratcher: the study doesn’t distinguish between academic surveys and academic experiments. It is challenging when looking at a single Qualtrics thing to know if it is a survey (everyone answers the same question, the goal is generalizability to a larger population) or an experiment (people are randomly assigned a treatment). Since it is often easier to make a case for the validity of an experiment on MTurk versus a survey, the constant use of the word ‘survey’ kind of bugs me. Example: ” nearly all of their (academic) requests (89%) consisted of surveys. “
Anyway, it is an interesting study for all of us to read!