On Peer Review

One of the distinctive properties of academic scholarship is that it is often subjected to “peer review.” “Peer review” generally means that a set of persons, who have well-documented relevant knowledge in the field of the scholarship, anonymously review the product of the scholarship. The review is critical, in the sense that reviewers are asked to assess whether the results are novel (thus, not redundant to prior work), whether they are carefully presented to be understandable by the targeted audience, and whether they are a nonnegligible addition to human knowledge/thought/accomplishment. Peer review is a feature of academic book production, journal publications, educational programs, and grant proposals; it is a key feature of tenure and promotion reviews for academics.

In today’s political environment in which grants to artists, writers, and scientists are scrutinized in unprecedented ways, there are some public misconceptions about peer review. One I’ve heard is that peer review is merely another label for a way that interest groups fund their friends. I read your work and judge it’s worthy for funding, and you reciprocate for my work.

This criticism doesn’t match my experience with Federal funding agencies and private foundations. A central feature of the peer review process is that the reviewers cannot have prior relationships with the proposer. It is routine that former students, previous academic colleagues, and current colleagues are eliminated from potential reviewers because of potential conflicts of interests. In review groups examining proposals from multiple units, with some of the work coming from units of those in the review group, members leave the discussion and cannot vote on work from their own institution.

Indeed, it is an ethical obligation of reviewers to refuse if asked to review work of someone tied to them by past work. They have the obligation to note that an unbiased review cannot be easily given.

Another counter-argument to the claim that peer review is a fancy word for funding one’s friends is the anonymity of the process. Most all publishers pledge never to reveal the identity of reviewers to the author(s) of the work being reviewed. Reviews are anonymized. This permits criticisms to be provided without any concern and without future implications of personal or professional relationships. Sometimes the author of the work being reviewed is not revealed to the reviewer, a double-blinding of the process.

Another misconception may arise from the word “peer.” In my experience, the process attempts to recruit reviewers who are the most knowledgeable in the relevant field. For some of the authors, especially those junior to the field, these can’t really be considered “peers.” They’re more easily considered established leaders of the relevant field. It is that they are deemed best suited for judging whether a new work breaks new and important ground. Perhaps, if we were starting from scratch on the nomenclature, we’d call it “expert review” not “peer review.”

The attempt to get the most accomplished to be reviewers leads to a glut of review requests for such persons. Indeed, one of the soft underbellies of the peer review process is the difficulty of recruiting reviewers to do the (typically unpaid) review work. Journal editors report contacting 10-15 academics to get 2-3 reviewers. The process is not perfect by any means. Indeed, the quality of the peer review process is totally dependent on the quality of the recruited reviewers. For example, at Georgetown we put a premium in promotion reviews on demonstrating the expertise of the reviewers as well as absorbing their evaluative comments.

Peer review is a deeply-held value in academic scholarship. It is not flawless, but it is laser-focused on quality in a way that many other evaluative processes in society are not. It’s a method worth defending, especially in these times of restricted resources of academic scholarship. It’s the academy’s way of assuring survival of only the fittest.