We all find ourselves talking often these days about what constitutes credible evidence relevant to any decision. This is especially common in discussions about the evaluation of scientific findings.
One feature of the scientific method that isn’t widely understood are the roles of reproducibility, replicability, and falsifiability in science. These three features are part of the disconnect between lay understandings of a “finding” or “evidence” and a scientist’s meaning of those words.
“Reproducibility” is sometimes meant that consistent results are obtained when a similar study is conducted. Some make a distinction between the notion of reproducibility and “replicability,” by which they mean, usually in an experimental setting, that identical results are obtained when exactly the same conditions occur again. In some sense, replicability requires more full allegiance to the original study’s conditions.
Clearly, scientists gain evidence of great value from events that are unique (e.g., naturally occurring but unique astronomical events). So some evidence can never really be duplicated. But in experimental and even in some observational settings, a conclusion is stronger when it can be reproduced in similar but slightly different settings. If there is a failure to achieve that, then science returns to the original finding to see if it can be replicated.
Part of the peer review process in scientific findings is attempting to judge whether the results are reproducible. Hence, scientists are required to describe their methods as fully as is feasible in order to judge the reproducibility of the results. Bad research can be winnowed out through this process.
Scientists are comfortable with the fact also that hypotheses can rarely be confirmed, but can be judged to be false. As many have noted, just because we have never seen a swan of a different color than white, doesn’t mean that all swans are white. Seeing one’s first black swan, however, immediately allows us to judge false the hypothesis that all swans are white.
Scientists are quite comfortable with the fact that the current evidence in any field is a cumulation of plausible hypotheses that have received some support. Careful scientists are quick to note the limitations of their knowledge (anticipating a falsification at some future point). They are comfortable when traditional findings are not reproduced. Often such failure to reproduce a result is the beginning of a breakthrough in the understanding of some phenomena. Falsification is big news that spurs on new work.
For the larger public, not embedded in the scientific method as a perspective, failure to reproduce a result or failure to replicate is troubling. Some viewed the original result as a fact, something that is immutable and fully true. When new scientific results show that the original finding is not true, or true under only a limited set of circumstances, they lose trust with the field that produced the result.
The growing concern with the lack of replicability and of reproducibility is related partially to this lack of understanding that a current scientific understanding is merely that – an understanding of the particular moment in the evolution of findings. At the same time, a field that rewards attempts to replicate and reproduce is one that can more efficiently focus on successful cumulative knowledge building.
I suspect that scientists can do better at communicating the nature of the scientific method. It’s key to maintaining the trust of the public.
Your father is a very wise man!
My father notes a difference between scientific mentality and lawyer mentality. In science, we observe evidence and then reach conclusions. But a lawyer starts with a position, then accumulates (or spins) evidence to fit. In our polarized society, we have too much lawyer-think, entrenched positions, no one really listening to the other side or the facts. Science is needed, not just to make your phone work, but as a mentality for honest debate about everything.
It is very difficult to achieve the benefits you describe of evidence based approaches in the policy field when the structure of government is designed to have people talk past one another.
Ps. I hope those Hoyas graduating this week have gotten this wisdom from all are faculty! Congrats to the class of 2017! Always evaluate and question and think outside the box! I am sure our convocation speaker today will address that in his talk about his life’s work.
Great discussion. We need to educate the public on the “facts” of scientific findings and that they are really ” findings” . Reminds me of the wise Med school professor who taught us to be lifetime learners as in twenty five years 25-50% of what we were taught as ” facts ” would be found out to be not true . Wise professor then and wise professor now encouraging us to educate the public on what we are doing . But there was also the other pearl taught about medicine that sometimes when you see your grandmother you KNOW. It’s your grandmother i.e. Sometimes intuition and gut reaction is very useful in medicine and life!