Profiling research units through user traces, and its consequences for research evaluation

The 2015 Altmetrics Workshop
Amsterdam, 9 October 2015

Ad Prins
Jack Spaapen


While there is an increasing interest in using Altmetrics for research evaluation, the value of such metrics for many SSH fields needs to be more researched. In SSH fields there are many research units with a variegated set of outputs, such as books, hybrid publications (or other forms of ‘grey’ literature), audio and video material, catalogues, etc., of which the impact is important but difficult to follow in traditional bibliometry. Fields and research units with such diverse types of output are interested in options to demonstrate the impact of their output through alternative qualitative and quantitative methods. In this paper we will show one such alternative, combining two methods for data collection and discuss the consequences for research evaluation. One method is based on Google Scholar as source for impact in the scientific domain. The other method, Contextual Response Analysis, is based on searches using a variety of internet search engines and of databases, identifying and classifying users of output in the broad social domain. Examples are taken from two large Dutch university research groups in the humanities and social sciences, also compared with results from analyses of two policy research institutes in The Netherlands. The results of these combined analyses lead to profiles of research units, based on user traces. Although these profiles have actively been used in actual research evaluations of the units, we note that the profiles not only differ from traditional metrics, but compel the evaluators to take a different approach to evaluation.

Supplementary materials