Considerations - RSP - Publishing Support - LibGuides at Coventry University

Once you are clear about the specific question you’re trying to address and the impact measuring that may have it's also important to consider various factors which may influence or skew a metric. The following highlights a number of factors which may need to be considered.

Career stage	The perceived 'prestige' of an author can influence the likelihood of being cited. It has also been suggested that those who have already accumulated a high number of citations, are likely to be cited again due to their prominence within a particular field. The h-index measures both the number of outputs an individual has produced (productivity) and number of citations they have received (influence). Consequently, early career researchers who are likely to have fewer publications, are unlikely to be properly represented by this metric.
Gender	Women on average publish less than men, are less likely to be listed as either first or last author and are less likely to be part of international collaborations (these tend to attract higher numbers of citations).
Discipline	Publication and citation practices vary across subject areas, and it is difficult to directly compare different disciplines. The type of research is also likely to affect citation rates e.g. 'pure' or applied, and venue of publication may also affect this.
Hyper authorship	Papers with large numbers of authors e.g. 100+, are likely to attract high numbers of citations. It is also sometimes difficult to determine an individual authors role. In some analysis tools, such as SciVal, you can remove these papers from the dataset to determine if they are having a distorting effect on results.
FTE	A part-time researcher is likely to produce fewer outputs than a full-time researcher within the same period. It is difficult to compare individuals within the same field and career stage without this additional information.
Language	Journals published in counties such as the United States, United Kingdom, Netherlands, France, Germany, Switzerland etc. are more likely to be indexed in Scopus and other Bibliographic databases, than smaller/developing countries or those publishing in a language other than English. Humanities research, in particular, may be affected by this in instances where research has a strong regional or national focus and may be published in journals with that audience in mind, consequently attracting fewer citations as a result.
Subjectivity	Peer review can be subjective and vary between reviewers leading to inconsistency. Unconscious/conscious bias also come into play in reviews; this can be mitigated in some instances by blind peer review.
Time frame	Citations take time to accrue, and disciplinary differences can affect this. It's difficult to assess impact of a paper using citations for those which are recently published. Topics may also change in popularity over time, affecting the number of citations they attract.

When using metrics, you should consider the five key questions below. These questions will help ensure you are using appropriate metrics to the questions being asked, and,also help ensure unintended consequences and biases outlined in previous sections are avoided.

1. Who does this discriminate against?

Should you take an alternative approach?

Are these limitations acknowledged?

2. How might this be gamed?

Can this be designed out?

3. Are there any unintended consequences?

Consider short/medium/long term effects - e.g. the short-term focus on one activity such as publishing in a high-ranking journal at the expense of disseminating research to the intended audience.