This new study talks about two people’s experience with using MTurk as Requesters (researchers). There’s some interesting stuff in here, first about what happens when technical issues cause Workers to not be able to submit studies etc. And what happens is you can get bad evals on Turkopticon. And then you get REquesters with hurt feelings.
“Believing in transparency, we used our real full names, rather than pseudonyms, for our AMT account, which appeared near our HITs. As a result, the angry emails we received and negative comments on Turkopticon felt to us like personal attacks. The negative reviews drew the attention of Six Silberman, who together with Lilly Irani created and manage Turkopticon . Silberman provided us with advice that was particularly meaningful and supportive: “Things can get really stressful in Turkland, with lots of people freaking out about accidents and assuming malicious intent at a moment’s notice.” A few Turkers we communicated with also sympathized with us: “If you have thick skin you can read up on the reactions of some of the people. This site [Turkopticon] is worker-controlled to fend off bad Requesters.” Silberman reminded us not to take the angry comments personally, but instead to respond clearly and courteously, and to communicate persistently. As an honest Requester, it was important to us to resolve Turker concerns before continuing with more tests and eventually running the study.”
I am a huge supporter of using ‘real names’ instead of cutesy pseudonyms because it puts a face on the Requester which I think translates to accountability. Would the attack seem less personal if your Requester name was “SciGal75”? Or are these researchers worried they got ‘outed’ as bad researchers? Part of me is saying ‘suck it up buttercup’. Honestly, an academic’s life is all about being attacked by people (student evals, anonymous reviews, etcetera). Listen to Six Silberman.
Cite: Brian McInnis and Gilly Leshed. 2016. Running user studies with crowd workers. <em>interactions</em> 23, 5 (August 2016), 50-53. DOI: http://dx.doi.org/10.1145/2968077