Social metrics for research: quantity and quality

altmetrics12
ACM Web Science Conference 2012 Workshop
Evanston, IL, 21 June 2012

William Gunn
william.gunn@mendeley.com
Jan Reichelt
jan.reichelt@mendeley.com
Mendeley

Abstract

Biomedical research is facing a crisis. Fewer than 10% of NIH grants are now getting funded (1), meanwhile an even smaller percentage of the research that does get funded turns out to be reproducible when another team tries to extend or apply the findings (2). Whatever the reason, it’s clear that we need better tools to understand what research is most likely to withstand the test of time and also to understand what research areas and teams can make the best use of research funds. New metrics based on social indicators such as bookmarks and shares on social media sites have been proposed as an addition to traditional methods involving citation counting. Mendeley provides one such source of metrics, which is a aggregation of the readership of academic papers, along with a layer of social and demographic metadata. Jason Priem has found that the readership metrics are significantly correlated (r=0.45) to Scopus citation counts (3).

Useful as these new metrics are, they tell only part of the story. It’s a useful bit of info to know the volume of citations or bookmarks or tweets about a paper or research field, but the real value lies in knowing what meaning the author intended to express when he linked paper A to paper B. Through the layer of social metadata collected around research objects at Mendeley we can start to address this challenge and add some quality to the quantitative metrics currently available.

References

1. http://www.niaid.nih.gov/researchfunding/paybud/pages/paylines.aspx

2. Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531-533. Retrieved from http://dx.doi.org/10.1038/483531a

3. Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web, 52900, 14. Digital Libraries; Physics and Society, . Retrieved from http://arxiv.org/abs/1205.5611

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

One Comment

  1. Posted June 14, 2012 at 7:57 pm | Permalink

    I agree that current altmetrics tell only part of the story; but I also think that’s true of any set of metrics. What we include and what we leave out when we measure things is a result of both technical capabilities and judgment. The resulting metrics are themselves rife with an interpretation of impact that is itself in need of interpretation.

    So, I question your claim that “the real value lies in knowing what meaning the author intended to express when he linked paper A to paper B” because I’m skeptical of the idea that we could ever get to “what the author intended to express” in any way that escapes the need for interpretation.

    Nevertheless, I’m anxious to see how you propose to enrich such interpretations!

One Trackback

  1. [...] but seldom cited while others are frequently cited but appear to have limited readership. In a recent presentation, William Gunn of Mendeley noted looking ahead that: Useful as these new metrics are, they tell only [...]

Post a Comment

Your email is never shared. Required fields are marked *

*
*