Researchers address ‘cargo cult’ science

A new article looks at some of the key criticisms of MTurk for organizational and information systems research and does a great job of taking some of the critiques that are written about all. the. time and turning them into advantages.

Critique: “We don’t know who these people are and people aren’t who they say they are.”

Advantage: A researcher can use multiple screeners to identify people who aren’t who they say they are (if they even exist) quickly and inexpensively.

Critique: Super Turkers distort results.

Advantages: Responses will be higher quality because people won’t be confused by questions, are attentive, and don’t get tired of answering surveys.

Critique: People lie.

Advantages: People always lie (ok this was from Dr. House, not from this article). Use of attention checks with appropriate punishments for failing attention checks (ie not getting paid) helps to identify people who aren’t good participants.

Critique: People on MTurk may not be normal.

Advantage: And people who complain about MTurk are (ok that was from me, not from this article).  The article says that paying at least minimum wage will help to attract more people to the site and to address any lack of normality.

It’s a good article and I encourage you to read it–I got the full piece from Researchgate.

Cite: Lowry, P. B., D’Arcy, J., Hammer, B., & Moody, G. D. (2016). “Cargo Cult” science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels. The Journal of Strategic Information Systems.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s