Long Term Worker Accuracy

A new study says that workers don’t exhibit satisficing behaviors over long periods. This is a great study with valuable findings.

“We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that workers fatigue
or satisfice over these long periods, producing lower quality
work. We find that, contrary to these claims, workers are ex-
tremely stable in their accuracy over the entire period. To
understand whether workers set their accuracy based on the
task’s requirements for acceptance, we then perform an ex-
periment where we vary the required accuracy for a large
crowdsourcing task. Workers did not adjust their accuracy
based on the acceptance threshold: workers who were above
the threshold continued working at their usual quality level,
and workers below the threshold self-selected themselves out
of the task.”

Hata, Kenji, Ranjay Krishna, Li Fei-Fei, and Michael S. Bernstein. “A Glimpse Far into the Future: Understanding Long-term Crowd Worker Accuracy.” arXiv preprint arXiv:1609.04855 (2016).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s