Attention please!

One of the biggest concerns among academics is that Workers participating in our experiments and surveys aren’t paying a lot of attention. As a result, it is important to include attention checks in your surveys.  Here are some to try:

1. If you are having Workers respond to a scale with multiple items, use of the items to instruct the participants which ‘button’ to check (strongly agree, neither agree or disagree, or the like).

2. On a unique screen, present a dense paragraph of information. In the paragraph, write something like “In the box below, regardless of what the question asks, write the word ‘spaghetti’.

3. Set a timer so the Worker is exposed to a page for a minimum amount of time. They’ll have nothing better to do so they’ll pay more attention to your questions (or not, they might be playing Words with Friends).

4. Present the same question twice, and reverse the wording the second time. Question 1 (with a ‘level of agreement to the statement’ scale): I know I can overcome whatever obstacles I encounter. Question 2: I question my ability to overcome whatever obstacles I encounter.

5. Warnings. In a study by researchers at MIT (although for some reason posted at <a href=”http://cess.nyu.edu/policon2012/wp-content/uploads/2012/02/berinsky_margolis_sances_screener_nyu.pdf”&gt; NYU </a>, a simple warning proved just as effective as screeners in response quality (and easier too!). Here’s the interesting result and the warning message:

The warning read: “We check responses carefully in order to make sure that people have read the instructions for the task and responded carefully.
We will only accept participants who clearly
demonstrate that they have read and understood the survey.
Again, there will be some very simple
questions in what follows that test whether you are reading the instructions. If you get these wrong, you will not be eligible for participation.”
Respondents who were given this additional warning had the answer
options I understand and I do not understand. While the survey did not explicitly screen out respondents who clicked I do not understand, only 1 respondent out of the 408 who received the warning responded as such.

Protip: don’t make warning messages too scary (that is, politely inform people why you’re using them).

One more protip: if you tell people you’re using attention checks, be sure you use at least one. Warning Workers and then not using any is just as impolite as being scary. Remember, Workers are people too. Treat them as respected members of your community.

If you have other ideas on attention checks–or insights on how frequently to use them, please post in the comments or email Kim (ksheehan@uoregon.edu).

3 thoughts on “Attention please!

  1. I’ve always intended to include warnings in my studies but haven’t gotten around to giving them a try. I’ll have to throw them in next time! As for #3, I use timers occasionally and they seem to work well, though I had one respondent type “Timers are garbage” where he was suppose to type the initials of his best friend… the other 199 respondents didn’t seem to have a problem with the timer, though.

    Thank you for the suggestions.

    Like

  2. Hmm, I wonder why the timer hate? Probably because they can’t move ahead as quickly as they would wish. How did you decide how long to set the timer for?

    Like

    1. Sorry for the late reply. I hadn’t noticed that you replied to me. I determined the time by going through the timer-utilizing stage of my study a few times myself to determine how long it would likely take people. Only one person seemed to be bothered by the timer, though, so I’m not too worried.

      I used a timer because participants weren’t required to complete everything on this particular page — however, if participants chose not to participate, I didn’t want it to be due them saving a few seconds of time.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s