Citation bait? Study about Academia.edu

This has nothing to do with MTurk but everything to do with academic publishing: Scholarly Kitchen’s report on a new study about Academia.edu.

I quote the report which quotes the study:

We find that a typical article posted to Academia.edu has 75% more citations than an article that is available elsewhere online through a non-Academia.edu venue: a personal homepage, departmental homepage, journal site, or any other online hosting venue.

And then Scholarly Kitchen EXPLODES OUR BRAINS (not really):

This paper suffers from a data problem. And that problem is their control group.

The researchers compared the performances of papers uploaded to Academia.edu to a control group–a random sample of papers selected from the same journals. If the randomization was successful, the control group should be similar in all respects to the Academia.edu group. In this way, differences observed in the data over time are likely attributable to Academia.edu and not some other cause.

If you look at their data (download the file: papers.csv.gz), you’ll notice something odd in the title of the first article: it is an errata and it belongs to the control group. Indeed, if you search the title list, you’ll find editorials, corrections, retraction notices, letters to the editor (and their responses), commentaries, book reviews, conference program abstracts, news, and even obituaries in the control group. As a general rule, these kinds of papers receive few (if any) citations. So, it’s no surprise that papers uploaded to Academia.edu outperformed a sample that included a good proportion of non-research material.

Data. So pesky.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s