One of the biggest concerns among academics is that Workers participating in our experiments and surveys aren’t paying a lot of attention. As a result, it is important to include attention checks in your surveys. Here are some to try:
1. If you are having Workers respond to a scale with multiple items, use of the items to instruct the participants which ‘button’ to check (strongly agree, neither agree or disagree, or the like).
2. On a unique screen, present a dense paragraph of information. In the paragraph, write something like “In the box below, regardless of what the question asks, write the word ‘spaghetti’.
3. Set a timer so the Worker is exposed to a page for a minimum amount of time. They’ll have nothing better to do so they’ll pay more attention to your questions (or not, they might be playing Words with Friends).
4. Present the same question twice, and reverse the wording the second time. Question 1 (with a ‘level of agreement to the statement’ scale): I know I can overcome whatever obstacles I encounter. Question 2: I question my ability to overcome whatever obstacles I encounter.
5. Warnings. In a study by researchers at MIT (although for some reason posted at <a href=”http://cess.nyu.edu/policon2012/wp-content/uploads/2012/02/berinsky_margolis_sances_screener_nyu.pdf”> NYU </a>, a simple warning proved just as effective as screeners in response quality (and easier too!). Here’s the interesting result and the warning message:
The warning read: “We check responses carefully in order to make sure that people have read the instructions for the task and responded carefully.We will only accept participants who clearlydemonstrate that they have read and understood the survey.Again, there will be some very simplequestions in what follows that test whether you are reading the instructions. If you get these wrong, you will not be eligible for participation.”Respondents who were given this additional warning had the answeroptions I understand and I do not understand. While the survey did not explicitly screen out respondents who clicked I do not understand, only 1 respondent out of the 408 who received the warning responded as such.
Protip: don’t make warning messages too scary (that is, politely inform people why you’re using them).
One more protip: if you tell people you’re using attention checks, be sure you use at least one. Warning Workers and then not using any is just as impolite as being scary. Remember, Workers are people too. Treat them as respected members of your community.
If you have other ideas on attention checks–or insights on how frequently to use them, please post in the comments or email Kim (email@example.com).