What Are Citation Metrics and Are They Relevant?

Quantifying the impact of research is a practice used to assess academic journals, researchers, and individual communications. In theory the higher the impact, the more important the work. Citation metrics are often used to assess candidates for academic positions or funding, and therefore researchers use journal-level metrics to decide the journals in which they want to publish their work. This article will dive into the definitions of certain citation metrics and their relevance in research.

The Metrics in Question

Citation metrics have three tiers: the article-level, concerned with the impact of the article; the author-level, concerned with the impact of the author; and the journal-level, concerned with the impact of the journal. These metrics are used to assess work from an institutional and a researcher perspective.

The Number of Citations

This can be a quite impressive statistic, for example when we have papers reaching over 300,000 citations (Oliver Lowry's Protein Measurement with the Folin phenol reagent.) However, this citation impact is not only a representation of the quality of the work, but other considerations, such as the number of researchers in a field or how quickly research is able to move from conception to publication in a given field. In the example of Lowry's work, his assay is applicable in a number of chemistry, biology, biochemistry, and clinical research efforts, making it extremely citable.

When authors are assessed on strictly the number of citations, it seems obvious that this metric includes not only the importance of their work, but also their career stage. Furthermore, these numbers can be skewed, as a number of citations also originate from self-citation. D. W. Aksnes, in his comprehensive macro-study of self-citation in Norway from 2003, aptly discusses the reasons for and against self-citation, describes the many subtleties in self-citation studies, and quantifies that self-citation accounts for 17-31% of citations depending on the field of study.

Journals may also be judged on their strict number of citations, but just as for researchers, this can be simply a reflection of the lifetime of the journal. Furthermore, journal impact used to be dependent on delivery of hard-copies to universities and libraries. Now, thanks to the internet, journals are much more accessible to researchers, leveling the playing field for journal access in the 21st century. Journal-level metrics are also subject to self-citation distortions.

Attempts to More Appropriately Compare Impact

As the strict number of citations is often representative of many factors convoluted with the scientific impact of the work in question, efforts to deconvolute these distortions and isolate a more direct impact factor have been made. Often, this is done by correcting the number of citations with another criteria, some examples are included in the table below:

Citation Metric	Level	Definition	Description
h-index	Author Level	Author has h publications that have been cited at least h times.	Meant to include contributions of productivity and citation impact
i10-index	Author Level	Number of publications with at least 10 citations.	Reflects the number of papers contributing to a field
FWCI (Field-Weighted Citation Index)	Author Level	Ratio of the total citations received to the average number of citations for a given field.	FWCI of 1 is average, FWCI < 1 is less cited, and FWCI > 1 is more cited.
Impact Factor	Journal Level	The impact factor for a given year is equal to the number of citations from that year divided by the number of publications in the previous two years.	Reflects the average number of citations of a given journal for the articles published in the two preceding years.

Plenty of variations of the h-index that account for the number of authors on a given paper, or the collaborative relationship between the cited author and the citing author have been proposed, yet we rarely come across these in our academic careers. The number of citations, i10-index, and h-index, however, are on every google scholar author profile.

The Main Criticism of Citation Metrics

As any researcher knows, citation metrics do not capture the complexity of factors contributing to an article's, author's, or journal's statistics. The main criticism is that citation metrics simply do not reflect what they are intended to reflect about the quality or relevance of a specific research study.

To illustrate this criticism more specifically, we turn to Dr. Björn Brembs' publication assessing the correlation of journal impact factor and research reliability. In this article, Brembs reports that higher impact factors are associated with larger errors in p-value reporting errors, higher likelihood of gene-naming errors, a higher incidence of overestimating likely true effect, and lower-quality crystallographic data. Further more, impact factors had no correlation with statistical power in neuroscience or randomization or blinding in animal studies. The main critique? Journal impact factors give very little indication of the quality of the science published in the journal.

Take-Aways

It is important to keep the criticisms of citation metrics in mind when moving through research careers. Assessing candidates for academic positions, and awarding financial support based on citation metrics may be incentivizing bias in our research by supporting the same line of reasoning and creating a barrier to new approaches.

While we may not shift the status quo overnight, it is nevertheless paramount that researchers at least understand the metrics by which they are assessing studies and works; peers and candidates; and journals and institutions. Keeping the definitions in mind will also bring awareness to the biases and distortions that are implicit in using citation metrics as a tool for assessing quality.

Do you think Citation Metrics are valuable assessment tools?

• Yes
• No