Comparing MTurk to a “Commercially Purchased General Household Panel”

A new study does exactly this. The panel is purchased through Qualtrics, which doesn’t really tell us where the panel came from as Qualtrics uses different companies to collect data. The study also didn’t put a ‘US only’ qualification on MTurk.

Some findings:

“The USA Regular panel members took more time to complete their responses than either of the two MTurk samples. This suggests that the MTurk respondents did not read the questions as thoroughly and were, in fact, speeding—potentially yielding lower quality data.” Or that the Qualtrics people were slow. And not used to answering surveys.

” Responses to all attention filter questions differed significantly (p < .001) across the three sample groups  (editorial note: the three groups are Qualtrics, MTurk US, MTurk non-us) indicating differences in attentiveness to survey instructions (with implications for data quality). The response pattern is similar for the two USA respondent groups, but the non-USA MTurk group deviated greatly. Moreover, the non-USA MTurk sample had the least percent of correct responses for all questions indicating that this group of respondents paid the least amount of attention to the questions (and thus would be expected to furnish the lowest quality of data).”

“Our results suggest that MTurk samples may be dominated by non-USA respondents, which may result in different sample characteristics, response patterns and data quality. This in turn can impact the substantive results and conclusions drawn from the research. Trading off cost (MTurk has an ease and cost advantage), the research must make an informed choice of an Internet online sample source.”

The article does suggest that screening questions can be put in, although these are difficult on MTurk (what?). All in all, this study seems to start with a hatred of MTurk and builds from there.

Citation: Smith S, Roster C, Golden L, Albaum G. A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples. Journal Of Business Research [serial online]. August 2016;69(8):3139-3148. Available from: Business Source Complete, Ipswich, MA. Accessed August 9, 2016.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s