A new study investigates different ways designers can get feedback on designs and find that Turkers provide good feedback and ideas:
“Among many findings, we found that a paid task market provides feedback
that contains more design suggestions and is longer and more positively-valenced. A Web forum provides more process- oriented feedback and the most responses without payment, while social networks provide more design suggestions. All
three crowds produced feedback of similar perceived quality.”
One quibble I have with this study is that it refers to Turkers as ‘financially driven’ which we know is not an accurate description of many workers (or to say another way, it is only a partial description of many workers). I can’t find anywhere in the study where they list the payment offered. The other two ‘crowds’ were designated as an ‘enjoyment crowd’ and a ‘social crowd’.
Yen, Y. C. G., Dow, S. P., Gerber, E., & Bailey, B. P. (2016, June). Social Network, Web Forum, or Task Market?: Comparing Different Crowd Genres for Design Feedback Exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (pp. 773-784). ACM.