Journal Impact Numbers

Environmental & Science Education, STEM, Nature of Science, History of Science

Ed Hessler

How to judge the merits of a scientific paper is the subject of Jonathan Jarry's OSS column titled What a Journal Impact Factor Is and Isn't.  

Jarry writes that "You may have heard of the impact factor. It is a number given to an academic journal (think Nature or The New England Journal of Medicine) which is often erroneously used a proxy for how good the papers it publishes really are. If the journal has a high impact factor, it must mean the research you will find within it is solid, goes the sentiment; if the number is low, be skeptical."

After a thorough discussion of what impact factors are, Jarry goes on to discuss their misuse. He closes by writing "Ultimately, how good a paper is can be difficult to evaluate by someone who is not an expert in the field, but a few rules of thumb can help us eliminate many of the worst offenders in the health sciences: the lack of a control group, a small number of participants, and research exclusively done on animals should always make us skeptical that the findings are real and will apply in humans."

The underlined emphasis is his which is a link to a previously published column on separating the proverbial wheat from the chaff. Below is the take-home message he wrote for that column.

- "Figuring out the trustworthiness and relevance of a scientific paper first requires identifying what kind of study it is (if it even is a study), which helps us know if the evidence is likely to be strong or weak

- "There are red flags that should reduce our trust in the evidence presented in a paper, such as the absence of a control group, a very small number of research participants, and a spotlighting of a positive secondary result when the main outcome the study was designed to look at was negative

- "Evaluating the worth of a paper can be helped by having many scientists look at it, which is why data detectives who spend their spare time denouncing bad papers are helpful, as well as websites like PubPeer and Retraction Watch"