A new study on Twitter engagement and diabetes > used Turkers to conduct a content analysis of tweets with the hashtag #diabetes.
To do the content analysis, they had each tweet coded for informational category (whether it was about people, an event, a success story, a failure story, etc) and sender description (person, organization, neither). They had each tweet coded by four different Turkers at a cost of 7 cents per tweet. It took on average about three and a half minutes to code a single tweet.
To show validity, they calculated the intraclass correlation coefficient for each topic and user types. Any topic or user type classification selected by two or more Turkers for a tweet was assigned to the Tweet. They found reliability was good or excellent with four coders.
I’m unfamiliar with the study by Hipp et al that they used as the framework (abstract here . It will be interesting to see if this method makes it way to the social sciences.