This resource will introduce you to the topic of responsible metrics. It supports the understanding and implementation of the University's Standard on Responsible Metrics which can be found here.
There are a number of benefits for the individual, a research group and the institution, in using metrics to support qualitative methods of assessing research activities.
For an individual, metrics can provide a useful insight into how their research may have been disseminated and used. They can provide a mechanism to benchmark against peers and support future applications for funding and progression. They can also help to identify key researchers and outputs, places to publish and potential collaborators within a particular field.
For a research group or faculty, metrics can be used to measure performance against similar research groups in other institutions. They can also help evaluate and develop strategies for research as a group and to identify potential recruitment and collaboration opportunities.
For an institution, metrics can be used to measure performance against corporate plans and targets. They can also be used to inform research strategies, to improve prestige, to support staff and student recruitment, and to identify funding opportunities.
However, metrics can be complex and the misuse of them can be damaging to both individuals and organisations. It is essential that the way in which metrics are used is clear, fair and transparent.
As outlined in the Metric Tide report, responsible metrics can be defined by the following key principles:
o Robustness – basing metrics on the best possible data in terms of accuracy and scope
o Humility – recognising that quantitative evaluation should support, but not supplant, qualitative, expert assessment
o Transparency – ensuring that those being evaluated can test and verify the results
o Diversity – accounting for variation by research field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
o Reflexivity – recognising and anticipating the systemic and potential effects of indicators, and updating them in response
The following sections in this resource are based on the SCOPE process developed by the INORMS Research Evaluation Working Group for evaluating responsibly. They are intended to help you understand some of the options available and the limitations of different approaches.
The first thing you should consider when thinking about using metrics is the specific question that you are trying to answer. What is it that you are interested in and value?
This is an important step to ensure that we measure what we truly value. It is essential to identify what it is about an individual's, a group's or an institution's research performance that you are interested in? This may change over time and differ from the interests and values of others. For example, if you are trying to establish a new research group as a centre of excellence, your focus may be on developing a critical mass in terms of volume of publications and income. However, if you are trying to evaluate the quality of research within an established team, focusing on peer review scores may be more valuable. Equally, if you are trying to identify which journal to submit to, you need to identify which are the most suitable for your research area, your intended audience and if they accept the type of article you wish to submit. Comparing journal metrics such as CiteScore or ABS rankings may help if you have multiple journals to choose from.
We cannot ignore external drivers such as University league tables, and we may want to know how we 'perform' using their values. However, we must be wary of basing evaluation on external drivers. These values may not line up with our internal values and consequently, we should not always base internal assessment on them, especially when looking at particular groups or subject areas where these measures may not be appropriate.
If you have been using metrics for a while to answer particular questions, consider whether there are alternative approaches available and if these would be more appropriate now. It is easy to fall into the trap of doing something in the same way it has always been done.
There may be sources of metrics that we can easily access or approaches for analysis which are easier to undertake but this can result in the 'streetlight effect'. This is a type of observational bias which occurs when people only search for something where it is easiest to look. Similarly, questions should never be retrofitted based on data we already have.
Metrics should not be applied as a proxy measure for the assessment of individual researchers or articles. Distinctions between article-based and journal-based metrics are importance and it is essential to recognise the limitations of each metric as we consider how best to answer the original question. There are many things that could be measured, but just because we can, doesn't mean we should!
Be SMART
When considering what your question is and whether metrics are appropriate, the SMART acronym may be helpful.
Remember
Be clear about the questions you are trying to answer and identify core values. Can these be measured? If so, identify relevant metrics that will support you in answering these questions.
Consider holding a Values Workshop. This may be particularly useful if you are looking at metrics from the research centre or institutional perspective. You can use a values workshop to:-
Review values in your Centre's/School's mission statement
Involve staff, both academic and professional services, to create a list of additional values
Identify top/core values
Can these be measured?
Individual metrics are not 'good' or 'bad' but you need to ensure they are appropriate to the question you are trying to answer. It is all dependent on what, who and how you are trying to measure, as well as the time frame under consideration.
As well as selecting appropriate measures, using metrics responsibility involves having an awareness of the impact that measuring something may have on behaviour. Metrics can be gamed in various ways and behaviour which would be desirable can end up being viewed as just 'compliance'.
Key things to consider are the discipline and the size/structure of the entity under evaluation. Think about:
There are lots of reasons why we might want to measure. These include:
The table below, created by Lizzie Gadd (2019), provides an indication of what level of caution should be applied when using metrics in different contexts, i.e. how much care we should take with any analysis and the weight of any results. It can be used as a starting point to consider 'how' and 'why' we are measuring and check that the indicator(s) used are appropriate to what you are seeking to measure.
When assessing individuals or small groups using metrics, the impact is likely to be much greater than when assessing larger entities where measuring is likely to have little affect. Consequently, care must be taken in terms of selecting the metrics used and how the outcomes are interpreted.
A change in behaviour requires more than measuring. As the saying goes 'you can't fatten a pig by weighing it'. In order for appraisal to be valuable, it is important to ensure you are using appropriate metrics to enable useful analysis which will then lead to action. Understanding, capability, opportunity and motivation are essential in developing benchmarks and measurements that lead to positive change.
Gadd, E. (2019) The blind and the elephant: bringing clarity to our conversations about responsible metrics. The Bibliomagician. May 15 2019. thebibliomagician.wordpress.com/2019/05/15/the-blind-and-the-elephant-bringing-clarity-to-our-conversations-about-responsible-metrics/
Options for measuring research can be both quantitative and qualitative. Often a combination of these approaches provides the most valuable insight. We should be wary of trying to measure qualitative things such as 'excellence' via quantitative indicators.
For example, the number of citations does not always correlate with the quality of an output. Similarly, the ratio of teachers to students doesn't provide any information on the quality of the teaching, although this is often used as a proxy in rankings.
Qualitative measures include peer review, which allows expert judgement of research rather than measurement by numbers. However, this has its own limitations. We all have conscious and unconscious biases which can affect judgement, and qualitative measures are by their nature subjective. We can reduce some of the limitations of peer review in a number of ways:
If possible you should engage with those who you are seeking to evaluate and discuss which options are appropriate before beginning.
Golden Rules
Below are a selection of some commonly used metrics. Remember all metrics have their limitations and a single metric will not fully demonstrate a journal or article's value or performance, see the benefits and limitations tab for more information.
Total number of outputs produced by an individual or group over a specified time period.
Publications with co-authors affiliated to international/national organisations or corporate bodies.
Number of times a publication is downloaded.
Total number of citations for an individual, output, group. There are a variety of reasons why an output may be cited including:
- Recognising leaders in subject field
- Giving credit to prior work which you are building on
- Use of a standard methodology/equipment
- Highlighting work which is not well known
- Disputing or criticising previous work
- Supporting ideas/claims and providing a background to the subject area.
These metrics can provide an indication of the 'prestige' of a journal by looking at the number/percent of publications within the top 1,5,10 or 25% of the most cited journals within the data source. This can be calculated in a number of ways including:
SNIP is a metric which intrinsically accounts for field-specific differences in citation practices by comparing each journal's citations per publication with the citation potential of its field. This allows direct comparison of journals in different fields. SNIP is calculated annually from SCOPUS data.
CiteScore provides a simple way of measuring the citation impact of publications. CiteScore is based on the number of citations to documents received by a journal in four years, divided by the number of the same document types published and indexed in Scopus in those same four years. The image below shows an example of the CiteScore calculation for 2020.
SJR is based on the concept of a transfer of prestige between journals via their citation links. SJR weights each citation to a journal by the SJR of the citing journal. It accounts for journal size by averaging across recent publications and is circulated annually. A single citation in life sciences would be low value in fields where large numbers of citations are the norm, whereas in arts and humanities a single citation would be high value. For a detailed description see this paper.
Number/percent of publications within the top 1,5,10 or 25% of the most cited publications within the data source.
A measure of productivity and citation impact, the H-Index aims to measure the cumulative impact and relevance of an individual's scientific research output. It looks at the number of citations and the number of papers which have been published. So, for example, to receive a h-index of 5 an author will have had to publish 5 papers with at least 5 citations each. To increase this number to 6, the author will have to have 6 papers with at least 6 citations each.
Ratio of citations received relative to the expected world average for the subject field, publication type and year.
Based on how often articles published in a particular journal during the previous two years (e.g. 2000 and 2001) were cited by articles published in a particular year (e.g. 2002). The higher the JIF, the more frequently articles in that journal are cited.
This measure is available from Web of Science, however currently the University does not subscribe to this service.
METRIC | BENEFITS | LIMITATIONS |
---|---|---|
Scholarly Output |
|
|
Collaboration |
|
|
Downloads |
|
|
Citation counts |
|
|
Publications in top journal percentiles (SNIP) |
|
|
Publications in top journal percentiles (CiteScore) |
|
|
Publications in top journal percentiles (SJR) |
|
|
Publications in top citation percentiles |
|
|
H index |
|
|
Field Weighted Citation Impact (FWCI) |
|
|
Journal impact Factor (JIF) |
|
|
Altmetrics are metrics and qualitative data that are complementary to traditional, citation-based metrics. They provide insight into how people are interacting with research online and can be used alongside other metrics to help demonstrate early engagement and discussions around research.
Can be used to indicate:
There are two main providers of altmetric data.
This is available on the Pure Portal. The metrics are represented by an 'Altmetric Donut' and the colours vary depending on which sources the output has received attention from. The central score is a weighted approximation of attention based on volume, sources and authors.
Some of the sources covered are listed below:
Further information on the colours and weighting is available here. Note, we do not subscribe to this service and therefore the detail available in Pure may be limited.
This is available on Scopus and the Pure Portal for outputs with a DOI. Metrics are represented by the 'Plum Print' which shows the categories below.
The five categories are:
More information on Plum X is available here. Note, we do not subscribe to this service and therefore the detail available in Pure may be limited.
Benefits | Limitations |
---|---|
Accumulate quicker than traditional metrics can indicate early attention | As with all metrics they don't tell the whole story |
Capture more diverse impacts | Potential to be gamed |
Apply to more output types | Bear in mind sentiment - attention isn't always positive |
Relies on existing identifiers e.g. DOI, URL, ISBN, many outputs still not included | |
Can be difficult to determine value |
Scopus is a multidisciplinary database containing abstracts and cited references of peer reviewed literature, including journal articles, books and conference proceedings. It contains more than 7 million items, with cited references back to 1970 and more than 16 million author profiles.
Information on the content coverage can be found here. You can also download the full list of journal and book titles included from this link.
How do I access Scopus?
Scopus is a subscription service which can accessed via this link - https://www.scopus.com/. Logging into Scopus is recommended even when on campus, this allows you to save searches and create alerts.
Metrics available in Scopus
SciVal
SciVal is an online tool which uses Scopus data from 1996 to present, in order to analyse research outputs, identify collaborative partnerships and see trends.
SciVal consists of 5 modules:
How do I access SciVal?
SciVal is a subscription service which can be accessed via this link - https://www.scival.com.
How to use SciVal
You can use SciVal to:
Metrics available in SciVal
There are over 30 different metrics available in SciVal. You should ensure you select appropriate metrics for your particular question and that you are aware of influencing factors such as size of group/number of outputs, discipline, time frame and coverage.
For a full range of metrics available in SciVal see the metrics guidebook.
Google Scholar provides an alternative citation count to Scopus and is openly available. You can search for articles or authors here. Citations to articles are computed and updated automatically as Google Scholar updates. Scholar trawls the web to pick up references to articles and does not require a source to meet the same criteria as Scopus in order to be included. This can result in there being a significant difference between your citation count in Google and on Scopus.
Google Scholar Metrics enable authors to quickly gauge the visibility and influence of recent articles in scholarly publications. Scholar Metrics can be used to browse the top 100 publications by their 5 year h-index and h-median metrics. Scholar Metrics are currently based on our index as it was in June 2020.
Further information on metrics in Google Scholar can be found here.
Why don't the metrics in Google Scholar look the same as those in Scopus?
Metrics such as citation counts for an individual author may differ significantly when looking at Scopus and Google Scholar. Google Scholar has a more inclusive approach and therefore may include more outputs for an individual, particularly if they are working in the social sciences/humanities area. Scopus has a set of criteria which all sources must meet to be indexed (more information here), consequently, this limits the number of outputs and citations included in the metrics provided. Google Scholar is also more prone to technical errors such as multiple records for a single document, incorrect/incomplete metadata and inclusion of non-scholarly outputs.
It's important to consider various factors which may influence or skew a metric. The following highlights a number of factors which may need to be considered.
The perceived 'prestige' of an author can influence the likelihood of being cited. It has also been suggested that those who have already accumulated a high number of citations, are likely to be cited again due to their prominence within a particular field.
The h-index measures both the number of outputs an individual has produced (productivity) and number of citations they have received (influence). Consequently, early career researchers who are likely to have fewer publications, are unlikely to be properly represented by this metric.
Women on average publish less than men, are less likely to be listed as either first or last author and are less likely to be part of international collaborations (these tend to attract higher numbers of citations).
Publication and citation practices vary across subject areas and it is difficult to directly compare different disciplines. The type of research is also likely to affect citation rates e.g. 'pure' or applied, and venue of publication may also affect this.
Papers with large numbers of authors e.g. 100+, are likely to attract high numbers of citations. It is also sometimes difficult to determine an individual authors role. In some analysis tools, such as SciVal, you can remove these papers from the dataset to determine if they are having a distorting effect on results.
A part time researcher is likely to produce fewer outputs than a full time researcher within the same time period. It is difficult to compare individuals within the same field and career stage without this additional information.
Journals published in counties such as the United States, United Kingdom, Netherlands, France, Germany, Switzerland etc. are more likely to be indexed in Scopus and other Bibliographic databases, than smaller/developing countries or those publishing in a language other than English.
Humanities research, in particular, may be affected by this in instances where research has a strong regional or national focus and may be published in journals with that audience in mind, consequently attracting fewer citations as a result.
Peer review can be subjective and vary between reviewers leading to inconsistency. Unconscious/conscious bias also come into play in reviews; this can be mitigated in some instances by blind peer review.
Citations take time to accrue and disciplinary differences can affect this. It's difficult to assess impact of a paper using citations for those which are recently published. Topics may also change in popularity over time, affecting the number of citations they attract.
When using metrics you should consider the five key questions below. These questions will help ensure you are using appropriate metrics to the questions being asked, and will also help ensure unintended consequences and biases outlined in previous sections are avoided.
1. Who does this discriminate against?
2. How might this be gamed?
3. Are there any unintended consequences?
4. Does the cost of measuring outweigh the benefits?
5. Does measuring research improve it?
Finally you should evaluate the results of any research metric use. This stage may overlap somewhat with the probe section, but it provides an opportunity for ongoing learning, assessing what worked and what could be improved and influencing processes and actions in the future.
To help familiarise yourself with some of the functionality in Scopus and SciVal if you have not used these tools previously there are a couple of simple practice exercises below.
For more information on Scopus and SciVal please see the links provided in this course to guidance documents, the pages on LibGuides or contact scival@coventry.ac.uk
A short video guiding you through each of these exercises is available below
For help with SciVal please contact: scival@coventry.ac.uk
Research & Scholarly Publications
FL320, Lanchester Library
Coventry University
Frederick Lanchester Building
Gosford Street
Coventry, United Kingdom
CV1 5DD
Telephone: 024 7765 7568
Email:
Open Access and Institutional Repository - oa.lib@coventry.ac.uk
Research Data Management - rdm.lib@coventry.ac.uk