Last week I lamented the pressure for fast decisions in this world overwhelmed by data but under-resourced for information. Here, “data” means a report of attributes of some person, organization, event, or other entity. “Information” occurs when data are used to help make decisions. It gives meaning to data.
In a few days, my attention moved to other issues, but it returned to the issue when I read about two recent biomedical papers, one in the Lancet and the New England Journal of Medicine (NEJM). The Lancet article contained data collected as late as April 14, 2020. The article was published a little more than a month later, May 22, 2020. The NEJM article was first published on May 1, 2020, based on data collected through the end of March, 2020. Its disclosure forms, often submitted with the manuscript are dated between March 30 and April 17, 2020. This is clearly 1) rapid research, and 2) rapid peer review of the product. Both journals later retracted the articles.
Peer review is a treasured aspect of science and academic research more generally (see here and here). Written work proposed for publication is examined by those disconnected from the creators. There is an attempt to remove any prejudgment of the quality of the work by the reviewer. Colleagues in the same institution are not permitted to make recommendations on publications. Prior collaborators generally cannot evaluate each other’s work. PhD mentors are not asked to comment on their protégé’s work.
At the same time, the reviewers are chosen for their knowledge of the domain in which the work is situated. They themselves should have contributed to important discoveries or developments of the field. They should be knowledgeable of current research/scholarship in the field. They should be aware of the diverse theoretical and empirical approaches taken in the field.
Their job is to critique the work. They are asked to find flaws in the methods used in the work. What was overlooked; what, underemphasized; what, overblown? To do so, they need full disclosure of what was done and how conclusions were reached.
In some technical fields the review of a manuscript (be it paper-length or book-length) can take months of careful review, if done properly. This is not a fast process. Reviews of nontechnical material can progress more quickly, but careful reading of anything in print is not a fast process.
The troubling aspect of new developments, often in scientific journals is the pressure to have quick turnarounds of the review. I have a friend who, in discussing alternative research designs to answer some question, often says, “Quick, well done, or cheap; pick two” in considering alternative research designs. Most consumers (and unfortunately, some funders of research) want all three attributes. A similar statement might apply to peer review. A fast peer review is often not a thorough, thoughtful peer review. Having a small number of reviewers is clearly cheaper of the resources of the journal, but limits the diversity of scrutiny to the work. Cheap and well done takes time.
In the case above, it appears that some of the reported features of the research, when carefully scrutinized were missed in the peer review step. Indeed, there are hints of questionable data that led the authors themselves to retract the NEJM article.
I have no doubt that the world now needs the fastest scientific discovery process possible. But I also completely understand the pressure to be the first to published a breakthrough discovery. However, peer review is stressed in these circumstances. The “death of slow” in this domain has severe costs.
So, what’s the fix? The National Academies panel on reproducibility and replicability recommended increasing the funding and immediacy of replication studies, whenever possible. The more surprising and impactful is the finding the more important that it be immediately subjected to replication attempts. In some domains, of physics findings are published without review and then subject to review.
If science is going to shrink the pre-publication peer review process, we need other protections from premature release of scientific findings.
The Lancet NEJM hydroxychloroquine debacle is a critical story of our time, thank you for posting. It is an important lesson. Most importantly, in the end, science got it right, and did so rather quickly. Multiple peer reviewed studies done “the right way” continued in spite of the hype, and three days ago FDA retracted it’s premature recommendations on hydroxychloroquine:
https://www.fda.gov/media/138945/download
Along the way, both journals retracted these papers, in very public fashion. One of the three authors of both papers, Amit Patel, has been fired from the Univ. of Utah, and the “source” of data for the papers, a private data analytics firm named “Surgisphere” has, for all intents and purposes, been drummed out of legitimate scientific circles, they will never be trusted again. Ever. All of this happened in a matter of days, against a forceful politically motivated (originating at the highest levels) push for using hydroxychloroquine vs COVID – 19. The fact that science got it right in a matter of days is heartening. More heartening still is that at least most physicians and other brave and dedicated people neck deep in the clinical trenches are listening astutely. Whether the politicians will or will not listen remains to be seen. Whether Lancet and NEJM have learned their lessons also remains to be seen, but most of us are optimistic. Several things went wrong, a number of key facts were not common knowledge until after the fact. If they were, the papers would likely not have been published. Among other points:
The private data analytics firm (not an academic group with a scientific reputation to protect, but a private profit motivated data analytics firm) providing the “data” for these studies was/is started and owned by the second of the three authors that appears on both papers, Dr. Sapan Desai. Patel and Desai are apparently related by marriage and Patel introduced Desai to the third co author of both papers, Mandeep Mehra, rumor has it, at Desai’s wedding. The personal relationships and clearly conflicted relationship between the academic authors and the data analytics firm providing the “data” should obviously have been made known much earlier. Then again, since no one I know has ever previously used data from “Surgisphere”, presumably Lancet and NEJM should’ve dug deeper. Surgisphere has not responded to a requested audit and none of the hospitals from which Desai’s Surgisphere supposedly mined the data used in the papers admit working with Surgisphere. Lancet and NEJM clearly screwed up, but this is not news for “elite” journals at the top of the publication pile.
All of this against the backdrop of intense political pressure coming from none other than the White House in the midst of a global pandemic that has claimed the lives of 400K + world wide and counting. A once – in – a – generation perfect storm for scientific fraud, but this is clearly not science as normally done. Best guess, more heads will roll at Lancet and NEJM, the anonymous reviewers of these papers will be (are) compromised, Surgisphere will (hopefully) suffer a quick and timely banishment from academic science, forever, and Desai Patel and Mehra will shuffle off the stage with irreparably damaged reputations now spiraling down a flushed toilet. This is all as it should be. It’s not what most scientists would prefer as a seasonal science story on the front pages, but it is a good one. In the aftermath, hopefully the press will dive into the more inspiring stories with equal vigor, but probably not, those tend not to sell as many newspapers or t.v. adverts. Nonetheless, thousands of dedicated basic scientists working 12 – 14 days worldwide are making substantial progress on vaccines and other therapies. Clinical scientists are doing trials the right way (it took them a week or two, but they got this right and even convinced the FDA to do the right thing). Agencies continue to support the basic science, and most importantly, the public continues to enroll in clinical trials and, in spite of intense political pressure to the contrary, support public health for the public good. Because they know that, rare examples of fraud notwithstanding, in the end science always gets it right.
“Quick, well done, or cheap; pick two” — I have a new motto!