We all find ourselves talking often these days about what constitutes credible evidence relevant to any decision. This is especially common in discussions about the evaluation of scientific findings.
One feature of the scientific method that isn’t widely understood are the roles of reproducibility, replicability, and falsifiability in science. These three features are part of the disconnect between lay understandings of a “finding” or “evidence” and a scientist’s meaning of those words.
“Reproducibility” is sometimes meant that consistent results are obtained when a similar study is conducted. Some make a distinction between the notion of reproducibility and “replicability,” by which they mean, usually in an experimental setting, that identical results are obtained when exactly the same conditions occur again. In some sense, replicability requires more full allegiance to the original study’s conditions.
Clearly, scientists gain evidence of great value from events that are unique (e.g., naturally occurring but unique astronomical events). So some evidence can never really be duplicated. But in experimental and even in some observational settings, a conclusion is stronger when it can be reproduced in similar but slightly different settings. If there is a failure to achieve that, then science returns to the original finding to see if it can be replicated.
Part of the peer review process in scientific findings is attempting to judge whether the results are reproducible. Hence, scientists are required to describe their methods as fully as is feasible in order to judge the reproducibility of the results. Bad research can be winnowed out through this process.
Scientists are comfortable with the fact also that hypotheses can rarely be confirmed, but can be judged to be false. As many have noted, just because we have never seen a swan of a different color than white, doesn’t mean that all swans are white. Seeing one’s first black swan, however, immediately allows us to judge false the hypothesis that all swans are white.
Scientists are quite comfortable with the fact that the current evidence in any field is a cumulation of plausible hypotheses that have received some support. Careful scientists are quick to note the limitations of their knowledge (anticipating a falsification at some future point). They are comfortable when traditional findings are not reproduced. Often such failure to reproduce a result is the beginning of a breakthrough in the understanding of some phenomena. Falsification is big news that spurs on new work.
For the larger public, not embedded in the scientific method as a perspective, failure to reproduce a result or failure to replicate is troubling. Some viewed the original result as a fact, something that is immutable and fully true. When new scientific results show that the original finding is not true, or true under only a limited set of circumstances, they lose trust with the field that produced the result.
The growing concern with the lack of replicability and of reproducibility is related partially to this lack of understanding that a current scientific understanding is merely that – an understanding of the particular moment in the evolution of findings. At the same time, a field that rewards attempts to replicate and reproduce is one that can more efficiently focus on successful cumulative knowledge building.
I suspect that scientists can do better at communicating the nature of the scientific method. It’s key to maintaining the trust of the public.