Coventry University has made a dedication to undertaking responsible metrics. In 2019, the university implemented a Responsible Metrics Standard aligned with the Leiden Manifesto. The implementation of this standard in 2019 demonstrated the University’s intention to engage with the responsible metrics agenda and commitment to ensure the responsible use of metrics across the Group. Following this, the university then signed the Declaration on Research Assessment (DORA) in order to further this commitment in line with the other HEIs
DORA recognises the need to improve the ways in which the outputs of scholarly research are evaluated. DORA’s vision is to advance practical and robust approaches to research assessment globally and across all scholarly disciplines. It is a worldwide initiative covering all scholarly disciplines and all key stakeholders including funders, publishers, professional societies, institutions, and researchers. For more information and to view current signatories see https://sfdora.org/
The overriding principle of DORA is to break the assumed link between certain metrics and research quality. It argues for qualitative judgments to be exercised, with research judged on its own merits. The key recommendations from DORA include:
Not using journal level metrics as a surrogate measure of individual article quality
Being clear about the criteria used in research evaluation
Considering the value of all output types
Using a range of metrics and incorporating qualitative indicators.
Challenging practices which rely on inappropriate metrics
The following guide should help prepare you to use research metrics appropriately and responsibly by taking you through all the steps required to evaluate your metrics use.
The first thing you should consider is the specific question that you are trying to answer. What is it that you are interested in and value?
It is essential to identify what it is about an individual's, a group's or an institution's research performance that you are interested in. This may change over time and differ from the interests and values of others. For example:
We cannot ignore external drivers such as University league tables, and we may want to know how we 'perform' using their values. However, we must be wary of basing evaluation on external drivers. These values may not line up with our internal values and consequently, we should not always base internal assessment on them, especially when looking at particular groups or subject areas where these measures may not be appropriate.
If you have been using metrics for a while to answer particular questions, consider whether there are alternative approaches available and if these would be more appropriate now. It is easy to fall into the trap of doing something in the same way it has always been done. There may be sources of metrics that are easier to access or approaches that are easier to undertake but this can result in the observational bias known as the 'streetlight effect' where people only search for something where it is easiest to look. Similarly, questions should never be retro-fitted based on data we already have.
Metrics should not be applied as a standard measure for the assessment of individual researchers or articles. Distinctions between metrics are important and it is essential to recognise the limitations of each metric as we consider how best to answer the original question.
There are many things that could be measured, but just because we can, doesn't mean we should!
When considering what your question is and whether metrics are appropriate, the SMART acronym may be helpful.
Be Specific
Can the value be Measured quantitatively?
What are you trying to Achieve?
Is it Robust i.e. will it provide the information you need to prompt further queries and to explain results?
What is the Timeframe under consideration?
Remember: Be clear about the questions you are trying to answer and identify core values. Can these be measured? If so, identify relevant metrics that will support you in answering these questions.
Consider holding a Values Workshop: This may be particularly useful if you are looking at metrics from the research centre or institutional perspective. You can use a values workshop to:
Individual metrics are not 'good' or 'bad' but you need to ensure they are appropriate to the question you are trying to answer. It is all dependent on what, who and how you are trying to measure, as well as the time frame under consideration.
As well as selecting appropriate measures, using metrics responsibility involves having an awareness of the impact measuring may have on behaviour. Metrics can be gamed in various ways and behaviour which would be desirable can end up being viewed as just 'compliance'.
Key things to consider are the discipline and the size/structure of the entity under evaluation. Think about:
There are lots of reasons why we might want to measure. These include:
The table below, created by Lizzie Gadd (2019), provides an indication of what level of caution should be applied when using metrics in different contexts, i.e. how much care we should take with any analysis and the weight of any results. It can be used as a starting point to consider 'how' and 'why' we are measuring and check that the indicator(s) used are appropriate to what you are seeking to measure.
When assessing individuals or small groups using metrics, the impact is likely to be much greater than when assessing larger entities where measuring is likely to have little affect. Consequently, care must be taken in terms of selecting the metrics used and how the outcomes are interpreted.
A change in behaviour requires more than measuring. As the saying goes 'you can't fatten a pig by weighing it'. In order for appraisal to be valuable, it is important to ensure you are using appropriate metrics to enable useful analysis which will then lead to action. Understanding, capability, opportunity and motivation are essential in developing benchmarks and measurements that lead to positive change.
Gadd, E. (2019) The blind and the elephant: bringing clarity to our conversations about responsible metrics. The Bibliomagician. May 15 2019. thebibliomagician.wordpress.com/2019/05/15/the-blind-and-the-elephant-bringing-clarity-to-our-conversations-about-responsible-metrics/
Options for measuring research can be both quantitative and qualitative. Often a combination of these approaches provides the most valuable insight. We should be wary of trying to measure qualitative things such as 'excellence' via quantitative indicators. For example, the number of citations does not always correlate with the quality of an output. Similarly, the ratio of teachers to students doesn't provide any information on the quality of the teaching, although this is often used as a proxy in rankings.
Qualitative measures include peer review, which allows expert judgement of research rather than measurement by numbers. However, this has its own limitations. We all have conscious and unconscious biases which can affect judgement, and qualitative measures are by their nature subjective. We can reduce some of the limitations of peer review in a number of ways:
If possible you should engage with those who you are seeking to evaluate and discuss which options are appropriate before beginning.
It's important to consider various factors which may influence or skew a metric. The following highlights a number of factors which may need to be considered.
Career stage |
The perceived 'prestige' of an author can influence the likelihood of being cited. It has also been suggested that those who have already accumulated a high number of citations, are likely to be cited again due to their prominence within a particular field. The h-index measures both the number of outputs an individual has produced (productivity) and number of citations they have received (influence). Consequently, early career researchers who are likely to have fewer publications, are unlikely to be properly represented by this metric. |
Gender | Women on average publish less than men, are less likely to be listed as either first or last author and are less likely to be part of international collaborations (these tend to attract higher numbers of citations). |
Discipline | Publication and citation practices vary across subject areas and it is difficult to directly compare different disciplines. The type of research is also likely to affect citation rates e.g. 'pure' or applied, and venue of publication may also affect this. |
Hyper authorship | Papers with large numbers of authors e.g. 100+, are likely to attract high numbers of citations. It is also sometimes difficult to determine an individual authors role. In some analysis tools, such as SciVal, you can remove these papers from the dataset to determine if they are having a distorting effect on results. |
FTE | A part time researcher is likely to produce fewer outputs than a full time researcher within the same time period. It is difficult to compare individuals within the same field and career stage without this additional information. |
Language | Journals published in counties such as the United States, United Kingdom, Netherlands, France, Germany, Switzerland etc. are more likely to be indexed in Scopus and other Bibliographic databases, than smaller/developing countries or those publishing in a language other than English. Humanities research, in particular, may be affected by this in instances where research has a strong regional or national focus and may be published in journals with that audience in mind, consequently attracting fewer citations as a result. |
Subjectivity | Peer review can be subjective and vary between reviewers leading to inconsistency. Unconscious/conscious bias also come into play in reviews; this can be mitigated in some instances by blind peer review. |
Time frame | Citations take time to accrue and disciplinary differences can affect this. It's difficult to assess impact of a paper using citations for those which are recently published. Topics may also change in popularity over time, affecting the number of citations they attract. |
When using metrics you should consider the five key questions below. These questions will help ensure you are using appropriate metrics to the questions being asked, and will also help ensure unintended consequences and biases outlined in previous sections are avoided.
1. Who does this discriminate against?
2. How might this be gamed?
3. Are there any unintended consequences?
4. Does the cost of measuring outweigh the benefits?
5. Does measuring research improve it?
Finally you should evaluate the results of any research metric use. This stage may overlap somewhat with the probe section, but it provides an opportunity for ongoing learning, assessing what worked and what could be improved and influencing processes and actions in the future.
FL320, Lanchester Library
Coventry University
Frederick Lanchester Building
Gosford Street
Coventry, United Kingdom
CV1 5DD
Open Access and Pure Deposits - oa.lib@coventry.ac.uk
Research Data Management - rdm.lib@coventry.ac.uk
024 7765 7568
We hold regular drop-ins at all of our Research Centres. To find the next one, please see the full calendar of our drop-ins.