Last week I lamented the pressure for fast decisions in this world overwhelmed by data but under-resourced for information. Here, “data” means a report of attributes of some person, organization, event, or other entity. “Information” occurs when data are used to help make decisions. It gives meaning to data.
In a few days, my attention moved to other issues, but it returned to the issue when I read about two recent biomedical papers, one in the Lancet and the New England Journal of Medicine (NEJM). The Lancet article contained data collected as late as April 14, 2020. The article was published a little more than a month later, May 22, 2020. The NEJM article was first published on May 1, 2020, based on data collected through the end of March, 2020. Its disclosure forms, often submitted with the manuscript are dated between March 30 and April 17, 2020. This is clearly 1) rapid research, and 2) rapid peer review of the product. Both journals later retracted the articles.
Peer review is a treasured aspect of science and academic research more generally (see here and here). Written work proposed for publication is examined by those disconnected from the creators. There is an attempt to remove any prejudgment of the quality of the work by the reviewer. Colleagues in the same institution are not permitted to make recommendations on publications. Prior collaborators generally cannot evaluate each other’s work. PhD mentors are not asked to comment on their protégé’s work.
At the same time, the reviewers are chosen for their knowledge of the domain in which the work is situated. They themselves should have contributed to important discoveries or developments of the field. They should be knowledgeable of current research/scholarship in the field. They should be aware of the diverse theoretical and empirical approaches taken in the field.
Their job is to critique the work. They are asked to find flaws in the methods used in the work. What was overlooked; what, underemphasized; what, overblown? To do so, they need full disclosure of what was done and how conclusions were reached.
In some technical fields the review of a manuscript (be it paper-length or book-length) can take months of careful review, if done properly. This is not a fast process. Reviews of nontechnical material can progress more quickly, but careful reading of anything in print is not a fast process.
The troubling aspect of new developments, often in scientific journals is the pressure to have quick turnarounds of the review. I have a friend who, in discussing alternative research designs to answer some question, often says, “Quick, well done, or cheap; pick two” in considering alternative research designs. Most consumers (and unfortunately, some funders of research) want all three attributes. A similar statement might apply to peer review. A fast peer review is often not a thorough, thoughtful peer review. Having a small number of reviewers is clearly cheaper of the resources of the journal, but limits the diversity of scrutiny to the work. Cheap and well done takes time.
In the case above, it appears that some of the reported features of the research, when carefully scrutinized were missed in the peer review step. Indeed, there are hints of questionable data that led the authors themselves to retract the NEJM article.
I have no doubt that the world now needs the fastest scientific discovery process possible. But I also completely understand the pressure to be the first to published a breakthrough discovery. However, peer review is stressed in these circumstances. The “death of slow” in this domain has severe costs.
So, what’s the fix? The National Academies panel on reproducibility and replicability recommended increasing the funding and immediacy of replication studies, whenever possible. The more surprising and impactful is the finding the more important that it be immediately subjected to replication attempts. In some domains, of physics findings are published without review and then subject to review.
If science is going to shrink the pre-publication peer review process, we need other protections from premature release of scientific findings.