What is your question?

The first thing you should consider when thinking about using metrics is the specific question that you are trying to answer.  What is it that you are interested in and value?

This is an important step to ensure that we measure what we truly value.  It is essential to identify what it is about an individual's, a group's or an institution's research performance that you are interested in?  This may change over time and differ from the interests and values of others. For example, if you are trying to establish a new research group as a centre of excellence, your focus may be on developing a critical mass in terms of volume of publications and income. However, if you are trying to evaluate the quality of research within an established team, focusing on peer review scores may be more valuable.  Equally, if you are trying to identify which journal to submit to, you need to identify which are the most suitable for your research area, your intended audience and if they accept the type of article you wish to submit. Comparing journal metrics such as CiteScore or ABS rankings may help if you have multiple journals to choose from.

We cannot ignore external drivers such as University league tables, and we may want to know how we 'perform' using their values. However, we must be wary of basing evaluation on external drivers. These values may not line up with our internal values and consequently, we should not always base internal assessment on them, especially when looking at particular groups or subject areas where these measures may not be appropriate.

If you have been using metrics for a while to answer particular questions, consider whether there are alternative approaches available and if these would be more appropriate now. It is easy to fall into the trap of doing something in the same way it has always been done. 

There may be sources of metrics that we can easily access or approaches for analysis which are easier to undertake but this can result in the 'streetlight effect'. This is a type of observational bias which occurs when people only search for something where it is easiest to look. Similarly, questions should never be retrofitted based on data we already have.
 

Metrics should not be applied as a proxy measure for the assessment of individual researchers or articles.  Distinctions between article-based and journal-based metrics are importance and it is essential to recognise the limitations of each metric as we consider how best to answer the original question. There are many things that could be measured, but just because we can, doesn't mean we should!

Be SMART
When considering what your question is and whether metrics are appropriate, the SMART acronym may be helpful. 

• Be Specific
• Can the value be Measured quantitatively?
• What are you trying to Achieve?
• Is it Robust i.e. will it provide the information you need to prompt further queries and to explain results?
• What is the Timeframe under consideration?

Remember

Be clear about the questions you are trying to answer and identify core values. Can these be measured? If so, identify relevant metrics that will support you in answering these questions.

Consider holding a Values Workshop. This may be particularly useful if you are looking at metrics from the research centre or institutional perspective. You can use a values workshop to:-

• Review values in your Centre's/School's mission statement

• Involve staff, both academic and professional services, to create a list of additional values

• Identify top/core values

• Can these be measured?

Context

Individual metrics are not 'good' or 'bad' but you need to ensure they are appropriate to the question you are trying to answer.  It is all dependent on what, who and how you are trying to measure, as well as the time frame under consideration. 

As well as selecting appropriate measures, using metrics responsibility involves having an awareness of the impact that measuring something may have on behaviour. Metrics can be gamed in various ways and behaviour which would be desirable can end up being viewed as just 'compliance'. 

Key things to consider are the discipline and the size/structure of the entity under evaluation. Think about:

• Who are you evaluating? (individual, group, institution...)
• Why are you evaluating?
• Do you need to evaluate at all?

There are lots of reasons why we might want to measure. These include:

1. To analyse - understand where we are 
2. To advocate - promote an entity be that an individual or the institution as a whole
3. To be accountable - set targets and ensure goals are met
4. To acclaim - compare one entity to another
5. To adapt - promote a change/incentivise
6. To allocate - rewarding successes

The table below, created by Lizzie Gadd (2019), provides an indication of what level of caution should be applied when using metrics in different contexts, i.e. how much care we should take with any analysis and the weight of any results. It can be used as a starting point to consider 'how' and 'why' we are measuring and check that the indicator(s) used are appropriate to what you are seeking to measure.

When assessing individuals or small groups using metrics, the impact is likely to be much greater than when assessing larger entities where measuring is likely to have little affect. Consequently, care must be taken in terms of selecting the metrics used and how the outcomes are interpreted. 

A change in behaviour requires more than measuring. As the saying goes 'you can't fatten a pig by weighing it'.  In order for appraisal to be valuable, it is important to ensure you are using appropriate metrics to enable useful analysis which will then lead to action.  Understanding, capability, opportunity and motivation are essential in developing benchmarks and measurements that lead to positive change.

Gadd, E. (2019) The blind and the elephant: bringing clarity to our conversations about responsible metrics. The Bibliomagician. May 15 2019. thebibliomagician.wordpress.com/2019/05/15/the-blind-and-the-elephant-bringing-clarity-to-our-conversations-about-responsible-metrics/ 

Options

Options for measuring research can be both quantitative and qualitative. Often a combination of these approaches provides the most valuable insight. We should be wary of trying to measure qualitative things such as 'excellence' via quantitative indicators. 

For example, the number of citations does not always correlate with the quality of an output. Similarly, the ratio of teachers to students doesn't provide any information on the quality of the teaching, although this is often used as a proxy in rankings.  

Qualitative measures include peer review, which allows expert judgement of research rather than measurement by numbers. However, this has its own limitations. We all have conscious and unconscious biases which can affect judgement, and qualitative measures are by their nature subjective. We can reduce some of the limitations of peer review in a number of ways:

• Being aware of unconscious bias and the impact that can have 
• Ensuring reviewers are experts in the particular subject
• Double blind reviews or an open review process
• Triangulate peer reviews with metrics where appropriate

If possible you should engage with those who you are seeking to evaluate and discuss which options are appropriate before beginning. 
 

Golden Rules

• Always use both quantitative and qualitative measures where possible
• Never rely on an individual metric to provide the whole picture
• Always cite your source and be transparent about how figures are derived
• Acknowledge limitations
• Use metrics appropriate to your question and the discipline being evaluated

Research Metrics

Below are a selection of some commonly used metrics. Remember all metrics have their limitations and a single metric will not fully demonstrate a journal or article's value or performance, see the benefits and limitations tab for more information.

Scholarly Output

Total number of outputs produced by an individual or group over a specified time period.

Collaboration

Publications with co-authors affiliated to international/national organisations or corporate bodies. 

Downloads

Number of times a publication is downloaded. 

Citation counts

Total number of citations for an individual, output, group. There are a variety of reasons why an output may be cited including:

- Recognising leaders in subject field 

- Giving credit to prior work which you are building on 

- Use of a standard methodology/equipment 

- Highlighting work which is not well known 

- Disputing or criticising previous work 

- Supporting ideas/claims and providing a background to the subject area. 

Publications in top journal percentiles

These metrics can provide an indication of the 'prestige' of a journal by looking at the number/percent of publications within the top 1,5,10 or 25% of the most cited journals within the data source.  This can be calculated in a number of ways including: 

• Source Normalised Impact per Paper (SNIP) 

SNIP is a metric which intrinsically accounts for field-specific differences in citation practices by comparing each journal's citations per publication with the citation potential of its field.  This allows direct comparison of journals in different fields. SNIP is calculated annually from SCOPUS data. 

• CiteScore

CiteScore provides a simple way of measuring the citation impact of publications.  CiteScore is based on the number of citations to documents received by a journal in four years, divided by the number of the same document types published and indexed in Scopus in those same four years. The image below shows an example of the CiteScore calculation for 2020.

• Scimago Journal Rank Indicator (SJR)

SJR is based on the concept of a transfer of prestige between journals via their citation links.  SJR weights each citation to a journal by the SJR of the citing journal.  It accounts for journal size by averaging across recent publications and is circulated annually. A single citation in life sciences would be low value in fields where large numbers of citations are the norm, whereas in arts and humanities a single citation would be high value. For a detailed description see this paper.

Publications in top citation percentiles

Number/percent of publications within the top 1,5,10 or 25% of the most cited publications within the data source. 

H index

A measure of productivity and citation impact, the H-Index aims to measure the cumulative impact and relevance of an individual's scientific research output. It looks at the number of citations and the number of papers which have been published. So, for example, to receive a h-index of 5 an author will have had to publish 5 papers with at least 5 citations each. To increase this number to 6, the author will have to have 6 papers with at least 6 citations each.

Field Weighted Citation Impact (FWCI)

Ratio of citations received relative to the expected world average for the subject field, publication type and year.

• An FWCI of 1 means that the output performs as expected for the global average.
• More than 1 means that the output is cited more than expected according to the global average. For example, 1.20 means 20% more cited than expected.
• Less than 1 means that the output is cited less than expected according to the global average.

Journal Impact Factor (JIF)

Based on how often articles published in a particular journal during the previous two years (e.g. 2000 and 2001) were cited by articles published in a particular year (e.g. 2002). The higher the JIF, the more frequently articles in that journal are cited.  

This measure is available from Web of Science, however currently the University does not subscribe to this service. 

Altmetrics

Altmetrics are metrics and qualitative data that are complementary to traditional, citation-based metrics. They provide insight into how people are interacting with research online and can be used alongside other metrics to help demonstrate early engagement and discussions around research.

​Can be used to indicate​:

• Attention​
• Dissemination​
• Influence

There are two main providers of altmetric data.

1. Altmetric

This is available on the Pure Portal. The metrics are represented by an  'Altmetric Donut' and the colours vary depending on which sources the output has received attention from. The central score is a weighted approximation of attention based on volume, sources and authors.​ 

Some of the sources covered are listed below: 

• Policy documents
• News/blogs
• Social media
• LinkedIn
• YouTube
• Patents

Further information on the colours and weighting is available here. Note, we do not subscribe to this service and therefore the detail available in Pure may be limited.

2. PlumX

This is available on  Scopus and the Pure Portal for outputs with a DOI. Metrics are represented by the 'Plum Print' which shows the categories below. 

The five categories are:

• Citations – Takes into account traditional citation indexes such as Scopus, as well as Clinical or Policy Citations
• Usage – Indications if an output is being read. e.g. clicks, downloads, views, library holdings, video plays
• Captures – Where an item has been marked for future use e.g. bookmarks, code forks, favourites, readers, watchers
• Mentions – Activities such as news articles or blog posts
• Social media - Includes Twitter, Facebook posts etc. that reference the research. Social Media can help measure attention

More information on Plum X is available here. Note, we do not subscribe to this service and therefore the detail available in Pure may be limited.
 

Sources of Research Metrics

Scopus

Scopus is a multidisciplinary database containing abstracts and cited references of peer reviewed literature, including journal articles, books and conference proceedings. It contains more than 7 million items, with cited references back to 1970 and more than 16 million author profiles. 

Information on the content coverage can be found here. You can also download the full list of journal and book titles included from this link.

How do I access Scopus?

Scopus is a subscription service which can accessed via this link - https://www.scopus.com/. Logging into Scopus is recommended even when on campus, this allows you to save searches and create alerts.

Metrics available in Scopus

• Journal metrics – CiteScore, SCImago Journal Rank (SJR) and Source Normalized Impact per Paper (SNIP)
• Article metrics – PlumX, citations and Field weighted citation impact
• Author metrics – h-index, citation overview

SciVal

SciVal is an online tool which uses Scopus data from 1996 to present, in order to analyse research outputs, identify collaborative partnerships and see trends.

SciVal consists of 5 modules:

1. Overview – High level overview of researchers, institutions and groups
2. Benchmarking – Comparisons between researchers, institutions and groups
3. Collaboration – Explore networks and identify potential collaborators
4. Trends – Investigate topics in depth
5. Reporting – Create reports tailored to specific needs

How do I access SciVal?

SciVal is a subscription service which can be accessed via this link - https://www.scival.com.

How to use SciVal

You can use SciVal to:

• Create groups of researchers
• Create publication sets
• Analyse research performance using a range of metrics across different disciplines

Metrics available in SciVal

There are over 30 different metrics available in SciVal. You should ensure you select appropriate metrics for your particular question and that you are aware of influencing factors such as size of group/number of outputs, discipline, time frame and coverage. 

For a full range of metrics available in SciVal see the metrics guidebook.

Google Scholar

Google Scholar provides an alternative citation count to Scopus and is openly available. You can search for articles or authors here. Citations to articles are computed and updated automatically as Google Scholar updates. Scholar trawls the web to pick up references to articles and does not require a source to meet the same criteria as Scopus in order to be included. This can result in there being a significant difference between your citation count in Google and on Scopus.

Google Scholar Metrics enable authors to quickly gauge the visibility and influence of recent articles in scholarly publications. Scholar Metrics can be used to browse the top 100 publications by their 5 year h-index and h-median metrics. Scholar Metrics are currently based on our index as it was in June 2020.

Further information on metrics in Google Scholar can be found here. 

Why don't the metrics in Google Scholar look the same as those in Scopus?

Metrics such as citation counts for an individual author may differ significantly when looking at Scopus and Google Scholar.  Google Scholar has a more inclusive approach and therefore may include more outputs for an individual, particularly if they are working in the social sciences/humanities area. Scopus has a set of criteria which all sources must meet to be indexed (more information here), consequently, this limits the number of outputs and citations included in the metrics provided. Google Scholar is also more prone to technical errors such as multiple records for a single document, incorrect/incomplete metadata and inclusion of non-scholarly outputs.

Considerations

It's important to consider various factors which may influence or skew a metric. The following highlights a number of factors which may need to be considered.

Career stage

The perceived 'prestige' of an author can influence the likelihood of being cited. It has also been suggested that those who have already accumulated a high number of citations, are likely to be cited again due to their prominence within a particular field. 

The h-index measures both the number of outputs an individual has produced (productivity) and number of citations they have received (influence). Consequently, early career researchers who are likely to have fewer publications, are unlikely to be properly represented by this metric. 
 

Gender

Women on average publish less than men, are less likely to be listed as either first or last author and are less likely to be part of international collaborations (these tend to attract higher numbers of citations). 
 

Discipline

Publication and citation practices vary across subject areas and it is difficult to directly compare different disciplines. The type of research is also likely to affect citation rates e.g. 'pure' or applied, and venue of publication may also affect this. 
 

Hyper authorship 

Papers with large numbers of authors e.g. 100+, are likely to attract high numbers of citations. It is also sometimes difficult to determine an individual authors role. In some analysis tools, such as SciVal, you can remove these papers from the dataset to determine if they are having a distorting effect on results. 
 

FTE

A part time researcher is likely to produce fewer outputs than a full time researcher within the same time period. It is difficult to compare individuals within the same field and career stage without this additional information. 
 

Language

Journals published in counties such as the United States, United Kingdom, Netherlands, France, Germany, Switzerland etc. are more likely to be indexed in Scopus and other Bibliographic databases, than smaller/developing countries or those publishing in a language other than English. 

Humanities research, in particular, may be affected by this in instances where research has a strong regional or national focus and may be published in journals with that audience in mind, consequently attracting fewer citations as a result. 
 

Subjectivity

Peer review can be subjective and vary between reviewers leading to inconsistency. Unconscious/conscious bias also come into play in reviews; this can be mitigated in some instances by blind peer review. 
 

Time frame 

Citations take time to accrue and disciplinary differences can affect this. It's difficult to assess impact of a paper using citations for those which are recently published. Topics may also change in popularity over time, affecting the number of citations they attract. 

Five Key Questions

When using metrics you should consider the five key questions below. These questions will help ensure you are using appropriate metrics to the questions being asked, and will also help ensure unintended consequences and biases outlined in previous sections are avoided.

1. Who does this discriminate against?

• Should you take an alternative approach?
• Are these limitations acknowledged?

2. How might this be gamed?

• Can this be designed out?

3. Are there any unintended consequences?

• Consider short/medium/long term effects - e.g. the short term focus on one activity such as publishing in a high ranking journal at the expense of disseminating research to the intended audience.
• Could this discouraging innovation, initiative and non traditional ways of carrying out research?
• Are there any local/systemic effects?

4. Does the cost of measuring outweigh the benefits?

• Is there value in this metric?
• Does the increase in burden on an individual/group lead to a reduction in the anticipated 'performance' increase?

5. Does measuring research improve it?

• What work will go into improving a metric if it's 'disappointing'?

Evaluate

Finally you should evaluate the results of any research metric use. This stage may overlap somewhat with the probe section, but it provides an opportunity for ongoing learning, assessing what worked and what could be improved and influencing processes and actions in the future.

Check List:

• Did you take the right approach for the entity being measured?
• What needs to be changed if this process is repeated?
• What worked well?
• Is this metric still appropriate?
• Was this method as fair and transparent as it could have been?
• Are new data sources available?
• Are new indicators/metrics available?
• Has the Institution's/Centre's mission/strategy changed?

Practice Exercises

To help familiarise yourself with some of the functionality in Scopus and SciVal if you have not used these tools previously there are a couple of simple practice exercises below. 

For more information on Scopus and SciVal please see the links provided in this course to guidance documents, the pages on LibGuides or contact scival@coventry.ac.uk

Scopus

http://www.scopus.com

1. Review metrics associated with an article:

• Find an article published recently, this could be your own, a colleague's or one identified from a keyword search for an area of interest.
• Display the document details for that article
• Select view all metrics
• Look at the citation benchmarking percentile. How does this output compare to others within the same subject area and of the same age and type?
• Review the article’s PlumX metrics. Does it have any interactions? If so which are strongest: usage, captures, citations, mentions or social media?

2. Browse and compare sources:

• Navigate to the sources page in Scopus
• Select subject area and select an area of interest e.g. insect science
• Filter the results to only show open access journals
• Find the journal with the highest SNIP value

A short video guiding you through each of these exercises is available below

SciVal

http://www.scival.com

1. View a summary of metrics for an institution:

• Go to the overview module
• Select the institutions and groups icon on the left
• Ensure Coventry University is selected and set the date range to 2016-2020
• Review the Scholarly output, FWCI, percentage of publications with an international co-author and percentage of publications with a corporate author

2. Compare institutions:

• Select the benchmarking module
• Select the institutions and groups icon on the left
• Search for another couple of universities and add these to the list to compare
• On the y-axis select ‘outputs in top citation percentiles’ under Cited. Select Field weighted, outputs in top 10%, shown as a percentage
• On the x-axis select publication year
• Review the change over time of the different institutions

For help with SciVal please contact: scival@coventry.ac.uk

Further Reading/Resources

Online: 

• Bibliomagician Blog: https://thebibliomagician.wordpress.com/
• Leiden Manifesto: https://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351
• Librarian Quick Reference Cards for Research Impact Metrics: https://libraryconnect.elsevier.com/metrics 
• Measuring your Research Impact: http://myri.conul.ie/ 
• Metrics toolkit: https://www.metrics-toolkit.org/ 
• San Francisco Declaration on Research Metrics (DORA): https://sfdora.org/
• Snowball metrics: https://www.snowballmetrics.com/ 
• UK Forum on Responsible Metrics: https://www.universitiesuk.ac.uk/policy-and-analysis/research-policy/open-science/Pages/forum-for-responsible-research-metrics.aspx

Books: 

• Biagiolo, M., (ed.) and Lippman, A. (ed.) (2020) Gaming the Metrics: Misconduct and Manipulation in Academic Research. Cambridge: MIT press
• Sugimoto, C., and Lariviere, V. (2018) Measuring Research: What everyone needs to know. Oxford: Oxford University Press
• Tattersall, A. (ed.) (2016) Altmetrics: A practical guide for librarians, researchers and academics. London: Facet Publishing 

Articles:

• Ali, M.J. (2021) 'Questioning the Impact of the Impact Factor. A Brief Review and Future Directions'. Seminars in Opthalmology. Available from http://dx.doi.org/10.1080/08820538.2021.1922713
• Costas, R., Zahedi, Z., and Wouters, P. (2015) 'Do "altmetrics" correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective'. Journal of the Association for Information Science and Technology 66 (10), 2003-2019. Available from http://dx.doi.org/10.1002/asi.23309
• De Rijcke, S., Wouters, P., Rushforth, A.D., Franssen, T.P., and Hammarfelt, B. (2016) 'Evaluation practices and effects of indicator use-a literature review'. Research Evaluation 25 (2), 161-169. Available from http://dx.doi.org/10.1093/reseval/rvv038
• Derrick, G.E., Haynes, A., Chapman, S., and Hall, W.D. (2011) 'The Association between Four Citation Metrics and Peer Rankings of Research Influence of Australian Researchers in Six Fields of Public Health' PLoS ONE 6 (4), e18521. Available from https://doi.org/10.1371/journal.pone.0018521
• Erikson, M.G., and Erlandson, P. (2014) 'A taxonomy of motives to cite'. Social Studies of Science 44, 625–637. Available from http://dx.doi.org/10.1177/0306312714522871.
• Ovseiko, P. V. et al. (2016) 'A global call for action to include gender in research impact assessment'. Health Research Policy and Systems 14 (1), 156–65. Available from http://dx.doi.org/10.1186/s12961-016-0126-z.