DORA and Responsible Metrics

Signatory of Dora Sticker Coventry University has made a dedication to undertaking responsible metrics. In 2019, the university implemented a Responsible Metrics Standard aligned with the Leiden Manifesto. The implementation of this standard in 2019 demonstrated the University’s intention to engage with the responsible metrics agenda and commitment to ensure the responsible use of metrics across the Group. Following this, the university then signed the Declaration on Research Assessment (DORA) in order to further this commitment in line with the other HEIs

DORA recognises the need to improve the ways in which the outputs of scholarly research are evaluated. DORA’s vision is to advance practical and robust approaches to research assessment globally and across all scholarly disciplines. It is a worldwide initiative covering all scholarly disciplines and all key stakeholders including funders, publishers, professional societies, institutions, and researchers. For more information and to view current signatories see https://sfdora.org/

The overriding principle of DORA is to break the assumed link between certain metrics and research quality. It argues for qualitative judgments to be exercised, with research judged on its own merits. The key recommendations from DORA include:

Not using journal level metrics as a surrogate measure of individual article quality
Being clear about the criteria used in research evaluation
Considering the value of all output types
Using a range of metrics and incorporating qualitative indicators.
Challenging practices which rely on inappropriate metrics

The following guide should help prepare you to use research metrics appropriately and responsibly by taking you through all the steps required to evaluate your metrics use.

What is your question?

The first thing you should consider is the specific question that you are trying to answer. What is it that you are interested in and value?

It is essential to identify what it is about an individual's, a group's or an institution's research performance that you are interested in. This may change over time and differ from the interests and values of others. For example:

If you are trying to establish a new research group as a centre of excellence, your focus may be on developing a critical mass in terms of volume of publications and income.
If you are trying to evaluate the quality of research within an established team, focusing on peer review scores may be more valuable.
Equally, if you are trying to identify which journal to submit to, you need to identify which are the most suitable for your research area, your intended audience and if they accept the type of article you wish to submit. Comparing journal metrics such as CiteScore or ABS rankings may help if you have multiple journals to choose from.

Lamppost We cannot ignore external drivers such as University league tables, and we may want to know how we 'perform' using their values. However, we must be wary of basing evaluation on external drivers. These values may not line up with our internal values and consequently, we should not always base internal assessment on them, especially when looking at particular groups or subject areas where these measures may not be appropriate.

If you have been using metrics for a while to answer particular questions, consider whether there are alternative approaches available and if these would be more appropriate now. It is easy to fall into the trap of doing something in the same way it has always been done. There may be sources of metrics that are easier to access or approaches that are easier to undertake but this can result in the observational bias known as the 'streetlight effect' where people only search for something where it is easiest to look. Similarly, questions should never be retro-fitted based on data we already have.

Metrics should not be applied as a standard measure for the assessment of individual researchers or articles. Distinctions between metrics are important and it is essential to recognise the limitations of each metric as we consider how best to answer the original question.

There are many things that could be measured, but just because we can, doesn't mean we should!

Be SMART

When considering what your question is and whether metrics are appropriate, the SMART acronym may be helpful.

Be Specific
Can the value be Measured quantitatively?
What are you trying to Achieve?
Is it Robust i.e. will it provide the information you need to prompt further queries and to explain results?
What is the Timeframe under consideration?

Remember: Be clear about the questions you are trying to answer and identify core values. Can these be measured? If so, identify relevant metrics that will support you in answering these questions.

Consider holding a Values Workshop: This may be particularly useful if you are looking at metrics from the research centre or institutional perspective. You can use a values workshop to:

Review values in your Centre's/School's mission statement
Involve staff, both academic and professional services, to create a list of additional values
Identify top/core values
Can these be measured?

Context

Individual metrics are not 'good' or 'bad' but you need to ensure they are appropriate to the question you are trying to answer. It is all dependent on what, who and how you are trying to measure, as well as the time frame under consideration.

As well as selecting appropriate measures, using metrics responsibility involves having an awareness of the impact measuring may have on behaviour. Metrics can be gamed in various ways and behaviour which would be desirable can end up being viewed as just 'compliance'.

Key things to consider are the discipline and the size/structure of the entity under evaluation. Think about:

Who are you evaluating? (individual, group, institution...)
Why are you evaluating?
Do you need to evaluate at all?

There are lots of reasons why we might want to measure. These include:

To analyse - understand where we are
To advocate - promote an entity be that an individual or the institution as a whole
To be accountable - set targets and ensure goals are met
To acclaim - compare one entity to another
To adapt - promote a change/incentivise
To allocate - rewarding successes

The table below, created by Lizzie Gadd (2019), provides an indication of what level of caution should be applied when using metrics in different contexts, i.e. how much care we should take with any analysis and the weight of any results. It can be used as a starting point to consider 'how' and 'why' we are measuring and check that the indicator(s) used are appropriate to what you are seeking to measure.

Diagram showing level of caution when using metrics in certain contexts

Pigs When assessing individuals or small groups using metrics, the impact is likely to be much greater than when assessing larger entities where measuring is likely to have little affect. Consequently, care must be taken in terms of selecting the metrics used and how the outcomes are interpreted.

A change in behaviour requires more than measuring. As the saying goes 'you can't fatten a pig by weighing it'. In order for appraisal to be valuable, it is important to ensure you are using appropriate metrics to enable useful analysis which will then lead to action. Understanding, capability, opportunity and motivation are essential in developing benchmarks and measurements that lead to positive change.

Gadd, E. (2019) The blind and the elephant: bringing clarity to our conversations about responsible metrics. The Bibliomagician. May 15 2019. thebibliomagician.wordpress.com/2019/05/15/the-blind-and-the-elephant-bringing-clarity-to-our-conversations-about-responsible-metrics/

Options

Options for measuring research can be both quantitative and qualitative. Often a combination of these approaches provides the most valuable insight. We should be wary of trying to measure qualitative things such as 'excellence' via quantitative indicators. For example, the number of citations does not always correlate with the quality of an output. Similarly, the ratio of teachers to students doesn't provide any information on the quality of the teaching, although this is often used as a proxy in rankings.

Qualitative measures include peer review, which allows expert judgement of research rather than measurement by numbers. However, this has its own limitations. We all have conscious and unconscious biases which can affect judgement, and qualitative measures are by their nature subjective. We can reduce some of the limitations of peer review in a number of ways:

Being aware of unconscious bias and the impact that can have
Ensuring reviewers are experts in the particular subject
Double blind reviews or an open review process
Triangulate peer reviews with metrics where appropriate

If possible you should engage with those who you are seeking to evaluate and discuss which options are appropriate before beginning.

Golden Rules

Always use both quantitative and qualitative measures where possible
Never rely on an individual metric to provide the whole picture
Always cite your source and be transparent about how figures are derived
Acknowledge limitations
Use metrics appropriate to your question and the discipline being evaluated

Considerations

It's important to consider various factors which may influence or skew a metric. The following highlights a number of factors which may need to be considered.

Career stage	The perceived 'prestige' of an author can influence the likelihood of being cited. It has also been suggested that those who have already accumulated a high number of citations, are likely to be cited again due to their prominence within a particular field. The h-index measures both the number of outputs an individual has produced (productivity) and number of citations they have received (influence). Consequently, early career researchers who are likely to have fewer publications, are unlikely to be properly represented by this metric.
Gender	Women on average publish less than men, are less likely to be listed as either first or last author and are less likely to be part of international collaborations (these tend to attract higher numbers of citations).
Discipline	Publication and citation practices vary across subject areas and it is difficult to directly compare different disciplines. The type of research is also likely to affect citation rates e.g. 'pure' or applied, and venue of publication may also affect this.
Hyper authorship	Papers with large numbers of authors e.g. 100+, are likely to attract high numbers of citations. It is also sometimes difficult to determine an individual authors role. In some analysis tools, such as SciVal, you can remove these papers from the dataset to determine if they are having a distorting effect on results.
FTE	A part time researcher is likely to produce fewer outputs than a full time researcher within the same time period. It is difficult to compare individuals within the same field and career stage without this additional information.
Language	Journals published in counties such as the United States, United Kingdom, Netherlands, France, Germany, Switzerland etc. are more likely to be indexed in Scopus and other Bibliographic databases, than smaller/developing countries or those publishing in a language other than English. Humanities research, in particular, may be affected by this in instances where research has a strong regional or national focus and may be published in journals with that audience in mind, consequently attracting fewer citations as a result.
Subjectivity	Peer review can be subjective and vary between reviewers leading to inconsistency. Unconscious/conscious bias also come into play in reviews; this can be mitigated in some instances by blind peer review.
Time frame	Citations take time to accrue and disciplinary differences can affect this. It's difficult to assess impact of a paper using citations for those which are recently published. Topics may also change in popularity over time, affecting the number of citations they attract.

Five Key Questions

When using metrics you should consider the five key questions below. These questions will help ensure you are using appropriate metrics to the questions being asked, and will also help ensure unintended consequences and biases outlined in previous sections are avoided.

1. Who does this discriminate against?

Should you take an alternative approach?
Are these limitations acknowledged?

2. How might this be gamed?

Can this be designed out?

3. Are there any unintended consequences?

Consider short/medium/long term effects - e.g. the short term focus on one activity such as publishing in a high ranking journal at the expense of disseminating research to the intended audience.
Could this discouraging innovation, initiative and non traditional ways of carrying out research?
Are there any local/systemic effects?

4. Does the cost of measuring outweigh the benefits?

Is there value in this metric?
Does the increase in burden on an individual/group lead to a reduction in the anticipated 'performance' increase?

5. Does measuring research improve it?

What work will go into improving a metric if it's 'disappointing'?

Evaluate

Finally you should evaluate the results of any research metric use. This stage may overlap somewhat with the probe section, but it provides an opportunity for ongoing learning, assessing what worked and what could be improved and influencing processes and actions in the future.

Check List:

Did you take the right approach for the entity being measured?
What needs to be changed if this process is repeated?
What worked well?
Is this metric still appropriate?
Was this method as fair and transparent as it could have been?
Are new data sources available?
Are new indicators/metrics available?
Has the Institution's/Centre's mission/strategy changed?

Research & Scholarly Publications Team: Good Metrics Practices

DORA and Responsible Metrics

What is your question?

Be SMART

Context

Options

Golden Rules

Considerations

Five Key Questions

Evaluate

Check List:

Further Reading/Resources

Online:

Books:

Articles:

Contact Us

📍 Where to find us:

✉️ Email:

📞 Phone:

📅 Drop-In Calendar:

Quick Links