MTurk: Examining the degree of ‘nontrivia’

As a researcher, I’m tired of hearing about ‘wonky data’ from Turkers, and here’s a study that shows that the Myth of the Wonky isn’t supported, particularly for Turkers from North America.

This study compared Turkers who identified as either from the US (of European descent) and  from India.  The researchers wanted 100 people in each group and anticipated a lot of bad data: they  oversampled given that they were expecting a “nontrivial amount” of people to have missing data or to not complete the survey properly. So they sampled until they had ‘at least 200 participants’ from each culture.

Interesting result in bold: my interpretation is that there was a ‘nontrivial amount’ of people from India who gave bad data and, at MOST, a ‘trivial amount’ of European Americans (it is unclear how the study screened for this though).

Participants and design. European American or Indian full-time employees were recruited through’s crowdsourcing website, Mechanical Turk© (Buhrmester, Kwang, & Gosling, 2011). They completed a short online survey with a payment of US$1. We set the location such that only people living in the United States or India can participate. Similar to Study 1, we aimed to recruit at least 100 participants from each culture. However, because we were also expecting that a nontrivial amount of participants might have missing data or might not complete the survey carefully, we continued the data collection until there were at least 200 participants from each culture. After screening out participants who had missing data and/or failed the attention check questions, it resulted in a total of 317 participants in the analysis, of whom 114 were Indians living in India (79 males and 34 females; Mage = 32.60, SDage = 9.05; work experience: M = 14.93 years, SD = 10.26) and 203 were European Americans living in the United States (97 males and 104 females; Mage = 36.15, SDage = 10.43; work experience: M = 11.16 years, SD = 11.93).5 There was a significant main effect of age such that older participants reported having stronger motivation in general, t = 3.40, p = .001. No significant main effects of gender or past work experience were found, Fs < 1.69, ps < .20. Because these demographic variables did not alter the pattern of the major findings, they were not included in the subsequent analyses.

Citation: Kung, F. Y., Kim, Y. H., Yang, D. Y. J., & Cheng, S. Y. (2016). The Role of Regulatory Fit in Framing Effective Negative Feedback Across Cultures.Journal of Cross-Cultural Psychology, 1, 17.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s