Options for measuring research - RSP - Publishing Support

Options for measuring research can be both quantitative and qualitative. Often a combination of these approaches provides the most valuable insight. We should be wary of trying to measure qualitative things such as 'excellence' via quantitative indicators. For example, the number of citations does not always correlate with the quality of an output. Similarly, the ratio of teachers to students doesn't provide any information on the quality of the teaching, although this is often used as a proxy in rankings.  

Qualitative measures include peer review, which allows expert judgement of research rather than measurement by numbers. However, this has its own limitations. We all have conscious and unconscious biases which can affect judgement, and qualitative measures are by their nature subjective. The limitations of peer review can be reduced by:

Being aware of unconscious bias and the impact that can have

Ensuring reviewers are experts in the particular subject

Using double blind reviews or an open review process

Triangulating peer reviews with metrics where appropriate

If possible, you should engage with those who you are seeking to evaluate and discuss which options are appropriate before beginning. 

Research metrics are measurements that are designed to evaluate research in its various forms and outputs. Metrics can be used to evaluate single articles, individual researchers, whole journals and entire institutions. In a way, research metrics tell the story of your research, looking at where it is published, who responds to it, how they respond, who is collaborating, where it is impacting, etc.; but each metrics has its own limitations. As such you need to use various metrics, both quantitative and qualitative, in order to tell the whole story.

You can access metrics for individuals, groups of researchers, publication sets and institutions in a number of ways, some of these are described here.

Golden Rules

Always use both quantitative and qualitative measures where possible
Never rely on an individual metric to provide the whole picture
Always cite your source and be transparent about how figures are derived
Acknowledge limitations
Use metrics appropriate to your question and the discipline being evaluated

METRIC	BENEFITS	LIMITATIONS
Scholarly Output - Total number of outputs produced by an individual or group over a specified time period.	Can be used for an individual, a group or for the institution as a whole	Measures productivity rather than quality In some disciplines where the predominant output type is not journal articles/conference proceedings this number will be lower as outputs such as books take longer to produce/publish
Collaboration - Publications with co-authors affiliated to international/national organisations or corporate bodies.	Can be used in combination with other metrics to give insight into whether collaboration enhances impact/influence	In some subject areas such as humanities single authorship is the norm Assumes connections are maintained long term
Downloads - Number of times a publication is downloaded.	Useful early indicator of outputs receiving attention before citations have time to accrue	Cannot infer usage
Citation counts - Total number of citations for an individual, output, group.	Can be useful to see who is citing work and indicate where it is having impact Easy to measure	Citation practice varies by discipline e.g. in arts citations are uncommon and if work is cited this is often negative Take time to accrue  Reviews and papers with new methodologies will have higher citation rates
Publications in top journal percentiles (SNIP) - Metric comparing each journal's citations per publication with the citation potential of its field.	Field normalised May be useful to identify journals to publish in	Assumes outputs in 'high quality' journals are high quality as well Should not be used to assess performance of individual publications Can be affected by outliers (e.g. a paper receiving a larger volume of citations because it describes a new method causes a spike in SNIP the year it is published and for a few years after)
Publications in top journal percentiles (CiteScore) - CiteScore is based on the number of citations documents received by a journal in four years, divided by the number of the same document types published and indexed in Scopus in those same four years.	May be useful to identify journals to publish in Field normalised when used to compare percentiles	Assumes outputs in 'high quality' journals are high quality as well Should not be used to assess performance of individual publications
Publications in top journal percentiles (SJR) - The average number of weighted citations received in a year, divided by the number of the same document types published in the previous three years. SJR weights each citation to a journal by the SJR of the citing journal. For a detailed description see this paper.	Field normalised May be useful to identify journals to publish in	Assumes outputs in 'high quality' journals are high quality as well Should not be used to assess performance of individual publications
Publications in top citation percentiles - Number/percent of publications within the top 1,5,10 or 25% of the most cited publications within the data source.	Useful to benchmark contributions to the most influential, highly cited publications within similar fields or disciplines	Need a large enough publication set to provide meaningful results Inherit the same limitations as citation counts
H index - A measure of the number of citations and the number of papers which have been published. e.g. to get a h-index of 5, an author will have had to publish 5 papers with at least 5 citations each. To reach 6, the author will need 6 papers with at least 6 citations each.	Easy to source and understand	Cannot be used to compare individuals, doesn't take into account career stage or publication timeline Doesn't take into account small numbers of highly cited outputs for an individual Only ever increases with age
Field Weighted Citation Impact (FWCI) - Ratio of citations received relative to the expected world average for the subject field, publication type and year, where 1 is the global average.	Field normalised Allows comparison with the world average in that subject	Need at least 100 publication for a meaningful number Should not be used to assess individuals Cannot be used for recent publications (less than one year old)
Journal impact Factor (JIF) - Based on how often articles published in a particular journal during the previous two years were cited by articles published in a particular year.	May give an indication of 'academic' impact of a journal	Short citation window Can be affected by outliers Not field weighted