New Research on Reviews

As most readers know, Amazon really cares about peer reviews except on MTurk. There are no ‘overt’ ratings (meaning there are no ‘five star’ Turkers or ‘two star’ Requesters) at the site. Requesters, though, can review and assess Turker reputations (via HITs completed and approval rates) and Turkers can check out Requesters at sites like Turkopticon (where five star Requesters do exist).

A new study (abstract here, Tech Crunch article here) shows that there is a difference between sites with one-way only ratings (where, in this case, the guests rated the accommodation) and where there are two-way ratings (guests rate accommodations and hosts rate guests). To quote the Tech Crunch article:

(The Boston University researchers) studied the same properties on TripAdvisor, which does not have bilateral reviews, and Airbnb, which does. The research found that on Airbnb, ratings for the same property were 14 percent higher than on TripAdvisor. This is a big difference when you consider that sales across industries nosedive by slight differences in star ratings. Customers are 3x less likely to book hotel rooms with a three-star rating than those with five stars, and for e-commerce stores, products with a four-star rating get 11.6x more orders than those with three.

What’s causing the discrepancy? It may be due to simple social pressure. In an open, bilateral review systems, many give good reviews to get good reviews or to look good. Consider the guest who gives a 5-star review for an Airbnb stay, despite the broken air conditioner and moldy fridge, because he didn’t want to receive a bad review in return.
Aware of this issue, Airbnb changed from publishing reviews as they were collected to revealing reviews once both parties had submitted (or 14 days after the trip ended). Yet this doesn’t account for a second social factor: People want to look good. In social settings in which people’s identities are not anonymous, people tend to shy away from saying bad things because they don’t want to be the one who seems like a constant complainer or never-ending nagger.

The study suggests ways for reviews to be more reflective of the actual experience and less based on social pressures. First, ratings must have standards, understood by both sides. Secondly, reviews can’t be retalitatory. Finally, the stigma against negative reviews must be reduced: we must see that negative reviews have value (because honestly? Learning you are a cheapskate on the Turkopticon is a huge motivator to increase what you’re paying for HITs. But Requesters need to go there to find out).

There have been many Turker voices trying to get Amazon to have more transparent and consistent ratings. Time for them to start paying attention.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s