The meaning of impact in altmetrics

The 2015 Altmetrics Workshop
Amsterdam, 9 October 2015

Kim Holmberg
Timothy D. Bowman
Fereshteh Didegah


Altmetrics have the potential to provide a more nuanced view of where and how research has made an impact because there is great diversity in the types and sources of altmetric events; research products can be shared and discussed on many different online platforms from blogs to Twitter and from Wikipedia to different reference managers and social network sites. In addition, the audiences generating altmetrics can be diverse as both scholars and the public can create the altmetric events by interacting with various types of research products. And as most types of altmetric events do not show the final usage of research, the influence drawn from using altmetric events may be intangible. The diversity in the sources of altmetrics, how they are created and the intangible influence of altmetric events raise several questions: (1) What exactly is being measured with altmetrics? (2) What do different altmetric events reflect? (2) What is the meaning behind the altmetric events? and (4) Is what is being captured and measured actually what researchers intend to measure?
Currently most altmetric events are captured by monitoring online messages for unique identifiers, like DOIs, handles, or URLs. This does, however, mean that only a fraction of the interactions related to the research products are captured. Typically only the first mention of a research product is identified and captured, while the whole conversation surrounding this event remains concealed in the vast flow of information because the identifiers are not there for the search algorithms to detect. For instance, the captured altmetric events (tweets) on Twitter are often just the dissemination of research article titles, without any indication of engagement (Haustein, et al., 2014). These tweet mentions may or may not have triggered lengthy conversations about the articles, but when focusing the data collection only on the unique identifiers, these conversations always remain undetected.
And what can actually be deduced from a mere mention of a research product online? There is typically some information captured about the account holder, but there is not always additional information available. For instance, an altmetric event does not provide information about whether the account holder has viewed the actual research product, whether the research product has had some type of societal impact—including scientific, cultural, economic, educational and environmental—on them, or whether it has caused the account holder to change their behavior in some way because of it.
When considering citations, a citation reflects the use of earlier work which makes citation impact more tangible and the typical assumption is that increased citations and use indicate more value and more impact of research. But can the same assumption be made for different altmetric events? Do more tweets indicate more impact of some type? Do more Wikipedia references reflect higher value? Do higher readership rates on Mendeley suggest higher value?
The goal with any research evaluation is to identify research that is of higher quality. To perform this evaluation, indicators of impact are used as a proxy for quality because quality is a difficult value to determine (especially at large scale). Traditionally citations were used as a proxy for impact and quality, where more citations indicated higher impact and higher quality. Citations are, however, created by other researchers and thus reflect both a level of use among other researchers and a level of scientific impact. But because the public, or a mix of both scholars and the public, create many altmetric events, it has been suggested that altmetrics may be able to reflect other types of societal impact including environmental, cultural, economic, and educational impact. But how can these types of impact be distinguished from each other when examining altmetric events? Is it possible to distinguish between impact types by examining the source of the altmetric events (such as the demographics of the specific population of users)?
This presentation will aim at igniting a lively debate about the meaning of altmetrics and the approaches one can take to investigate the events, with special focus on the meaning and representations of impact.

Supplementary materials