Searching for the Best

Some recent reading evoked memories of various evaluative panels that I’ve experienced in the past few years. Some were created to award a national or international honor to the awardee, based on self- and/or others’ nomination. Some were review panels of interdisciplinary grants. Some were fellowship offers to promising mid-career scholars.

The panel members were often chosen from among those with established reputations in their field. Most often the panelists represented different disciplines. The assigned job of the panel was to identify the “best” among a set of proposed awardees. Sometimes the focus of the evaluation was the awardee; sometimes, the proposed work of the awardee. Inevitably, it became some blend of the two. That is, part of the evaluation of the promise of the work was whether the work could indeed be accomplished well by the awardee.

Lamont, in her book with the wonderful title of How Professors Think, notes that such panels execute their work with a complicated and jointly negotiated pattern of evaluative viewpoints. Sometimes they are focused on the “generalizability of the work.” Sometimes they are focused on its ability to evoke an “understanding of broader processes.” Sometimes, they value the deeper understanding that results from a “particular interpretation.”

These evaluative viewpoints are differentially susceptible to the so-called “Matthew effect,” which gives the more established line of thinking a distinct advantage in an evaluation. That is, when examining a proposed project, ideas that build upon years of successful knowledge construction in a field, have an advantage. They benefit from reflected glory of widely-accepted knowledge and technique. They receive the immediate respect of association with the current leaders of a field.

One might suspect that when the evaluators are focused on the generalizability of the proposed project, they might evoke their knowledge of what past scholarship has demonstrated in a field. When, in contrast, the panelists are highly valuing the creation of a particular interpretation of a topic, they might be less susceptible to the Matthew effect. Clearly, the cognitive bias associated with the Matthew effect threatens support of the new, the controversial, the different. In panels that I remember, such proposals receive the label of “high-risk.” (Even in such cases, however, the Matthew effect is present with a little twist – if the novel proposal is made by a famous, established scholar, the panel will tend to label it as “innovative” rather than “risky.”)

At various points of evolution of a field, it seems likely that generalizability becomes more important than understanding or novel interpretation. For example, one of the key issues facing educational research (e.g., what teaching techniques are more effective) was the domination of the field of small-scale experimental interventions that could not be replicated. That is, a project that showed great learning leaps in one classroom, implemented by one teacher, showed no such gains when later implemented in another classroom. The field began to call for greater attention to the generalizability of scholarship funded.

Some have noted that generalizability, understanding of basic mechanisms, and novel interpretation, can themselves be viewed as specific to particular uses of knowledge. The three might array themselves on a dimension of theory-application. Generalizability is an important attribute in applied knowledge. Novel interpretations are important in the generative step of understanding. Understanding basic processes are the building blocks of theories that later lead to generalizability.

The Matthew effect, giving advantage to building on large bodies of existing knowledge, is a natural bias that mitigates risk in applications. It’s probably deadly, however, for the giving value to the unexpected, the novel, the innovative. In short, thinking back over my panel memberships, sometimes the group should have explicitly warmed to the safety of the well-established and sometimes, to the new.