How does MTurk compare to Prolific Academic, Crowdflower and the Carnegie Mellon poll (also known as CBDR)?
CBDR was the ‘control’, so to speak, and comparing MTurk to PA and CF shows:
“In two studies, we found that participants on both platforms (PA and CF) were more naïve and less dishonest compared to MTurk participants. Across the three platforms, CF provided the best response rate, but CF participants failed more attention-check questions and did not reproduce known effects replicated on ProA and MTurk. Moreover, ProA participants produced data quality that was higher than CF’s and comparable to MTurk’s. ProA and CF participants were also much more diverse than participants from MTurk.”
So it looks like a thumbs up to Prolific Academic.
Nice job from some very expert MTurk researchers!
Eyal Peer, Laura Brandimarte, Sonam Samat, Alessandro Acquisti, Beyond the Turk: Alternative platforms for crowdsourcing behavioral research, Journal of Experimental Social Psychology, Volume 70, May 2017, Pages 153-163, ISSN 0022-1031, http://doi.org/10.1016/j.jesp.2017.01.006. (http://www.sciencedirect.com/science/article/pii/S0022103116303201)