Putting Scientometrics 2.0 in its Place [v0]

This is version 0 of an abstract to be presented at altmetrics11.

Ralph Schroeder
Lucy Power
Eric T. Meyer
Oxford Internet Institute

Measuring  science,  scientometrics,  has  provided  an  enhanced  understanding  of  how  science  is  changing  for  some time, and is thus useful for (among other things) research policy. With the rapid spread of Web 2.0 tools  in scholarly communication, it thus seems appropriate to apply scientometrics to Web 2.0 uses in science. Web  2.0  can  be  defined  here  as  ‘participatory  information  sharing’,  taken  from  Wikipedia  (http://en.wikipedia.org/wiki/Web_2.0,  accessed  3.3.2011),  which  in  this  case  is  enabled  by  ‘off‐the‐shelf’  tools  such  that  non‐  computer  experts  can  contribute  content  onto  the  Web  in  a  way  that  is  accessible  to  others.  The  popularity  of  Web  2.0  among  scholars  is  the  starting  point  for  the  argument  by  Priem  and  Hemminger  (2010)  for  a  ‘Scientometrics  2.0’.  They  argue  that,  on  the  one  hand,  researchers  are  nowadays  overloaded with content, and on the other that Web 2.0 offers powerful new measures of scientific outputs. A  related argument has recently been made by Evans and Foster (2011), who also start with the flood of digital  knowledge  that  researchers  are  nowadays  confronted  with.  They  go  on  to  sketch  what  they  call  ‘metaknowledge’, by which they mean the use of computational tools to identify patterns in how knowledge is  created in order to better master this process. This goal, of improving the evaluation, filtering, and mapping of  knowledge, is also the impetus behind the new programme outlined by Priem and Hemminger.

This  essay  argues  that  both of  these  new tool‐based  ideas  about  science  make  important  points  about  how  science is changing, but they overlook several features about the context of knowledge production. Briefly, the  overload that researchers are experiencing is something that they are bound to adapt to, following the law of  small numbers and the limits of the attention space (Collins 1998; Meyer and Schroeder 2009), features that  have  governed  and  continue  to  govern  science,  including  science  2.0.  Secondly,  it  is  true  that  Web  2.0  technologies  lend  themselves  to  powerful  new  ways  of  understanding  changes  in  research,  but  these  must  always be put in the context that web 2.0 tools are complementary to and add to existing modes of scholarly  communication. In short, scientometrics, including the 2.0 version, and like any account of knowledge, needs  to be put into a social context.

These points can be expanded briefly: From a sociological perspective, the structure of knowledge is controlled  by the limits of the attention space and the law of small numbers. Applied to online material, a researcher may  pay  attention  to  the  complementary  material  that  is  available  on  the  user‐generated  (or  ‘participatory  information sharing’) Web. However, the stream of research, while constantly growing, is also constantly being  winnowed – if only because much research is never cited and only a small amount of research contributes to  the advance of knowledge. Even if researchers may thus spend additional time and effort on new channels of  scientific  dissemination  on  an  everyday  basis,  ultimately  they  will  be  paying  attention  to  the  same  small  number  of  researchers  that  are  generating  novel  results  or  ideas.  This  also  applies  to  the  tools  for  ‘metaknowledge’, which need to be judged in the end by how they contribute to these new results.

Put  the  other  way  around,  what  scientometrics  2.0  tools  will  be  able  to  tell  us  is  how  the  new  channels  of  scientific dissemination are supporting this focus on those dominating the attention space. A useful question  here  will  be:  do  those  dominating  the  online  attention  space  differ  from  those  who  dominate  the  attention  space  in  the  channels  of  traditional  scholarly  dissemination?    We  would  expect  that  although  in  certain  respects,  new  ‘stars’  emerge  online,  the  main  impact  of  Web  2.0  will  be  to  make  their  traditionally  disseminated work also garner more attention. Further, only work that is assessed by peer review will enter  into  ‘winnowed’  knowledge,  thus  making  Web  2.0  dissemination  ancillary.  The  exceptions  are  new  online  channels  that  may  also  have  new  mechanisms  for  evaluation,  though  these  channels  –  like  individual  researchers – will compete with existing ones.  A  different  way  to  make  this  point  is  that  not  only  just  as  much,  but  more  material  will  now  receive  no  attention  at  all.  There  are  also  limits  to  ‘open’  peer  review  or  online  comments  or  recommendations  of  articles:  these  will  be  an  important  complement  to  existing  anonymous  reviews  and  anonymous  or  non‐ anonymous  comments:  yet  ultimately,  the  constraints  of  science  are  that  most  scientific  work  needs  to  be  evaluated  anonymously  (even  if  some  open  peer  review  journals  provide  an  additional  avenue  for  dissemination) while it is still knowledge‐in‐the‐making, and authorship needs to be asserted for knowledge‐ already‐made.

Thus  it  is  inevitable  that  there  will  be  an  ever  greater  diversity  –  a  proliferation  ‐  of  the  means  of  scientific  communication,  as  well  as  an  increasing  amount  of  knowledge  content.  Metaknowledge  can  be  useful  in  filtering  this  flood,  but  these  tools  will  also  add  to  the  cognitive  workload  (they  cannot  replace  consulting  sources of original scholarship, but are complementary) and they will also be subject to the laws of the limited  attention  space  and  small  numbers  (which  could,  in  fact,  be  built  into  metaknowledge  tools  –  producing  an  infinitely expandable capacity for meta‐ metaknowledges). It is also inescapable that the amount and diversity  of  Web  2.0  dissemination  of  research  will  add  complexity  to  evaluating  the  impact  of  research,  which  will  inevitably grow because of the increasing pressures on limited funding for research. This impact evaluation will  make use of scientometrics 2.0, but there is a limit here too: the limit that only a certain amount of research  can be said to have impact without making the very notion of impact meaningless.

Scientometrics  2.0,  like  other  online  content,  will  make  scholars  more  reflexive  about  their  visibility  and  impact. Collins argued some time ago that researchers are drowning in paper. Now we are drowning in even  greater  amounts  of  digital  resources.  Even  so,  the  problem  of  this  flood  will  be  solved  –  not  just  at  the  individual  level,  but  also  at  the  level  of  the  system  of  scholarly  communication:  only  a  few  researchers  and  institutions  concentrated  in  a  global  hierarchy  will  be  visible.  Even  as  the  total  number  of  researchers  and  outputs  expands,  this  expansion  is  reduced  by  specialization  and  winnowing.  Larger  and  more  long‐term  changes,  such  as  the  relative  decline  of  the  US  and  Europe  and  the  rise  of  countries  like  China  and  India  in  terms of research rankings, will affect this overall picture. Here, too, scientometrics 2.0 and metaknowledge  could be useful: do certain disciplines or geographical areas dominate the attention space of online knowledge  more than those of more traditional forms of disseminating knowledge? And where do these new means of  dissemination become complementary to established forms of scientific communication in becoming accepted  as  ‘legitimate’?  Scientometrics  2.0  and  metaknowledge  will  thus  complexify  the  research  landscape  rather  than simplifying it.  Yet this greater complexity also enriches our understanding of how knowledge works, even  if  it  also  puts  additional  demands  on  this  understanding.  The  full  version  of  this  paper  will  illustrate  these  arguments  by  reference  to  various  Web  2.0  tools,  including  scholarly  Wikis,  new  multimedia  life  science  journals, and data about Web 2.0 uses in the humanities and in physics.


Collins,  R.  1998.  The  Sociology  of  Philosophies:  A  global  theory  of  intellectual  change,  Cambridge:  Harvard  University Press.  Evans, J. and Foster, J. 2011. Metaknowledge. Science, vol. 331, no. 6018: 721‐725

Meyer,  E.T.,  Schroeder,  R.  (2009).  Untangling  the  Web  of  e‐Research:  Towards  a  Sociology  of  Online  Knowledge.  Journal of Informetrics 3:246‐260.

Priem, J. and Hemminger, B. 2010. Scientometrics 2.0: Toward new metrics of scholarly impact on the social  Web. First Monday, vol.15, no.7,  http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2874/2570


  1. Posted May 9, 2011 at 1:20 pm | Permalink

    Unless you did that already, It could be interesting that you establish and describe the concept of “personal strategies” of research, dissemination and reputation (as opposed to institutional/traditional). These personal strategies could be described as belonging to the non-traditional forms of knowledge dissemination, and as relying on the “off-the-shelf” Web 2.0 tools (Wikis, Twitter, Facebook, Blogs…). These strategies are often decided and managed at the level of the individual, within a given social context, in liaison and in full natural complementarity with other traditional forms of dissemination. The context of the individual ties it all together. You already coined this behavior when you wrote: “We would expect that although in certain respects, new ‘stars’ emerge online, the main impact of Web 2.0 will be to make their traditionally disseminated work also garner more attention.” I would be interested to discuss this further with you.

  2. ralph.schroeder
    Posted May 9, 2011 at 6:56 pm | Permalink

    thanks for this – and agree that there are personal strategies, though the point you raise then leads to at least two further ones: one concerns when personal strategies must ‘bend to the will’ of instutional/traditional forces (in this, the means of research dissemination)? And secondly, when will the new Web 2.0 forms of dissemination themselves take their fully institutionalized form or become traditional?! This just expands on your point, but alsoasks how much scope individuals now have while the new system is still in flux. Thanks for the comment, and look further to discussing it further. See you in Koblenz, Ralph

  3. Victoria Uren
    Posted May 19, 2011 at 4:29 pm | Permalink


    I am interested in your comment about “dominating the online attention space”. Are you aware of Huberman’s work on social attention online? there was an excellent keynote at EKAW last year
    (http://ekaw2010.inesc-id.pt/invited-speakers.html) but no slides seem to be available. His work is quite commercially oriented (turning attention on websites into profit) but could possibly be relevant for measuring attention to science. regards Victoria

  4. ralph.schroeder
    Posted May 20, 2011 at 8:08 am | Permalink

    this is a useful comment. For science, I borrow ‘attention space’ from Collins (as per the reference in our paper). I think you are right to point out that, for our argument, one would need to have an overall assessment of ‘scarcity’ (this is where economics perhaps comes in) of the total amount of attention online and offline. Huberman is useful here, but when I look at his publications, such as his article (with Wu) on ‘Novelty and Collection’ (2007), he has a psychologistic understanding of attention. I think we need a more sociological one, but I admit that this is still lacking. Thanks for pushing me to explore this further, and look forward to discussing it more, best wishes, Ralph

  5. Posted June 1, 2015 at 6:01 pm | Permalink

    how do we define iacpmt/quality in an operational way?I think that’s a really important question! A lot of discussions in this space do end up a bit confused thanks to different understandings of these terms. I generally feel like quality is a subjective judgement, while iacpmt is a bit more objective, since it’s referring to how an article (or dataset or whatever) has somehow changed something(s). Of course, there are many sorts of iacpmts, and a lot of things research can change: medical treatments, government policies, public opinion, disciplinary boundaries, lexicons, experimental methods, teaching approaches, theoretical stances, and of course much more. Not all of these are equally valuable to everyone; heck, some iacpmts might even be negative for some folks.So, I think there are a lot of ways to operationalize iacpmt, depending on what you care about. I think that’s not often been as clear as it should be in policy, funding, and tenure decisions. I’m really excited to see this starting to change, to see growing recognition that iacpmt can’t be just one thing.The exciting thing about altmetrics is that we can start maybe finding ways to measure iacpmts that were before unmeasurable. Not everyone is going to care about iacpmt on public education, for example. But some funders and admins, it turns out, do care. For them, citations from Wikipedia, science blogs, or mainstream media may be valuable indicators. Not everyone is going to care about immediate iacpmts on informal scholarly discourse. That’s cool. But for those that do (and there good reasons to, since this informal discourse often drives scholarship more than formal publication), tracking the blogs and Twitter feeds of selected scholars may prove a useful indicator. And there are tons of other cool potential indicators of various iacpmts out there, many of which have been studied in the scientometrics community for a while now, and some of which are pretty new: citations, patents, clinical guidelines, acknowledgements, Nobels, newspaper articles, doctoral committee membership, collaborations, grant awards, conference acceptances, reference manager inclusion, downloads, hires, hyperlinks, social bookmarking, inclusion in syllabi and many more.The challenge for, and promise of, altmetrics is to start using all these indicators, together, to tell this broader and finer-grained story of multiple iacpmts. To do that, we’re going to need a lot more social science describing the properties of these newer indicators, assessing their validity and examining their value. I’m excited that a lot of that’s starting to appear (in the recently-published , for example), and excited about continuing to push it forward.In the meantime, these metrics and the systems that gather them are helping to provoke thoughtful discussions about what we mean by iacpmt(s), and I think that’s a really positive thing.

Post a Comment

Your email is never shared. Required fields are marked *