Rethinking science in the context of the reproducibility crisis.

In 2015 a crisis of reproducibility left the scientific community in a state of disorientation very similar to the one health officials found themselves during  COVID in pre-vaccine times. This article inspired by my own experience in the lab and the mentioned reproducibility crisis was originally posted in 2018 and provides a framework to understand science anomalies pointing to a convergence of science and Jewish sources.

Rethinking Science in the context of the reproducibility crisis

In light of the current reproducibility crisis, this article proposes a different way to look at what scientific contributions represent. It proposes that the requirement for coherence in biological scientific publications masks the ambiguity and indetermination inherent to the practice of science. Recognition of that ambiguity is important as it allows for the coexistence of contrasting perspectives, and the development of mutually complementary models. As the publication process tends to filter out what does not fit into a linear logical narrative, it builds an artificial self-constructed sense of certainty. Instead, scientific publications should be assumed as useful particular ways of structuring order allowing for a multiplicity of viewpoints consistent with experimental observation.

“The Aleph, the only place on earth where all places are — seen from every angle, each standing clear, without any confusion or blending.”

“To signify the godhead, one Persian speaks of a bird that somehow is all birds; Alanus de Insulis, of a sphere whose center is everywhere and circumference, is nowhere; Ezekiel, of a four-faced angel who at one and the same time moves east and west, north and south.”

“I saw the Aleph from every point and angle, and in the Aleph I saw the earth and in the earth the Aleph and in the Aleph the earth; I saw my own face and my own bowels; I saw your face; and I felt dizzy and wept, for my eyes had seen that secret and conjectured object whose name is common to all men but which no man has looked upon — the unimaginable universe.”                                                                                                Jorge Luis Borges, The Aleph (1)

In recent years a crisis of reproducibility has put the scientific community in a state of perplexity. In response to this crisis, much emphasis has been put in the performance of research with rigor. Certainly, there are cases of research conducted in a sloppy manner, and cases of misconduct,  but even if one could clear out the scientific literature of those cases there would still be an intrinsic problem with the way science is practiced, how the scientific system is organized, and the limitation of science to deal with ambiguity. The root of the problem should be looked at not in what is published but in what it is not.

One of the pillars on which experimental science stands on is the idea that hypotheses are either true or false, that the scientific method can discern the truthfulness of any hypothesis, and if wrong, correct itself over time. On the basis of that assumption, two seemingly incompatible hypotheses can not be true at the same time. One has to be true and the other false and that assumption sets the basis for the whole ethical system that governs the practice of science.

In reality science reductionist approach makes researchers operate on the basis  of partial information that  reveal partial trues, sometimes contradictory,  reflection of a higher truth

In some instances, these partial trues can be reconciled with new experimental information leading to a unifying model, but in other cases, they can not.  Imagine you are a scientist trying to answer the following question: “Can I park my car in the 3rd floor of Boston Logan Airport’s terminal B Garage tonight at 9 PM?”. To answer the question you do the following experiment: you drive your car to the sited place at 9 PM and when you get through the ramp to the third floor you look to your right-hand side and see a red light sign that says “Full”.  You conclude that it is not possible to park the car in the said place and time and publish accordingly. Now a second scientist comes behind you and performs a different experiment.  Instead of looking to the right, looks to the left and sees an empty parking spot. He or she arrives to the opposite conclusion, that it is possible to park the car at the said place and time.  An argument is established, the second scientist probably experiencing a self-imposed or community ( i.e. reviewers) imposed higher barrier to publish.  Imagine now a  third scientist that being more thoughtful performed both experiments. In light of the contradicting data  the scientist will find the results non-conclusive and will not publish them. Ambiguity has been masked.

Eventually evaluating some other piece of information, i.e in the cultural context of where the garage is situated, what has more weight a sign saying you can not park or the physical availability of an empty parking spot, a decision is made to park the car or continue to the fourth floor, ignoring one of the discrepant pieces of information.   A choice is made, leaving behind the ambiguity and indetermination that allowed for that choice.

There is indeterminacy underlying the way in which scientific knowledge is generated.  Even though ambiguity presents itself in the lab on daily basis, it is consciously or unconsciously disregarded. The end-product of the discovery process, the published paper, is linear and logical but the discovery process is chaotic and involves many conscious or unconscious decisions on the way that affect the end result.  Each of those decisions sets a  deterministic course in an otherwise indeterministic landscape. Perhaps the decision involves the use of protocol A instead of protocol B, or just the use of protocol A without considering other options. We may assign to ambiguous results some rational explanation for instance sample variability.  To test a hypothesis proof is commonly sought by three, four or five different techniques. Often one of those techniques produces a result that differs from the rest, and as we expect results to be consistent,  we assume that something went wrong with the discrepant technique,  In doing so we mask ambiguity.

Reviewers and journals require self coherent stories and scientists produce them generating a self-constructed certainty that leaves out anything that does not fit into a linear narrative.  In the face of ambiguity scientists either do not publish as results are deemed non-conclusive,  or are biased in what they publish or in the way results are interpreted. Fortunately, this is something that began to change, with an increasing number of journals now accepting the inclusion of negative data.  However, negative data,  rather than being regarded as an accident should be seen as the natural manifestation of the ambiguous nature of science.

Shedding light with Light

Probably the first and best-established example in science in which one phenomenon is described by two alternative models is light.  So let’s shed light with light. Light can be described both as a wave and as a particle. The model describing light’s behavior as a wave has yielded applications like the photovoltaic cell,  solar energy, or x-ray crystallography.  The model describing light as a particle yielded X-ray imaging.  The experiments that led to the description of light as a particle however do not fit the narrative that explains light’s behavior as a wave and vice versa. The arrival to the coexistence of two alternative models has not been easy and without fights. It took more than a century to reach that point.

The indeterminacy underlying the dual behavior of light is likely, in light of the current reproducibility crisis,  a more common phenomenon in science than we think,  only that in our scientific way of thinking we are conditioned to disregard it,   in favor of a self-conceived certainty.

Paradigm shifts and Experimental Biology

Paradigm shifts are seen as the result of science self-correction over time. When the reproducibility crisis started, NIH published in its web page a statement saying   “science corrects itself over time, but that is not longer true in the short run”,  An alternative view is that that paradigm shifts are the manifestation at different time points of otherwise coexisting truthful alternative models,  theories or perspectives. With a larger scientific enterprise today than a century or two ago, data that supports alternative views are now produced synchronously.  Perhaps because it is an older discipline than experimental biology and is both experimental and theoretical, Physics has reached a point that allows for alternative models or theories coexistence. Examples are Newtonian and Relative Physics,  light’s dual wave/particle behavior, or theories in the process of validation like those of  Multiverse or String Theories.

The time seems to have come now for experimental biology to transit through a similar path.

A number of replication efforts conducted in a very controlled manner revealed reproducibility problems of published experiments in psychology and Cancer biology.  In psychology, a replication study on an effect called “verbal overshadowing” the pooled results of  22 groups succeeded in replicating results of a previously doubted study, but replication was sensitive to protocol parameters (2). In cancer, a  highly cited study published in Cancer Cell on a mutation in the gene IDH whose product can be detected in blood and potentially be used as a leukemia biomarker,  failed to replicate in a first batch of studies but succeeded in a more recent round, factoring in that  20% of leukemia patients carry the mutation (3) . A second study, however, on BET Inhibitors succeeded in replicating the in-vitro cell experiments but failed to replicate the in-vivo mouse experiments (4). Four other studies in the context of the Cancer Biology Reproducibility Project have also revealed reproducibility problems of previously published results or their statistical significance (5,6,7,8).    A poll conducted by the journal Nature with  1500 scientists revealed that ~70% of the scientists polled in the fields of biology and medicine failed to reproduce someone else’s experiments, and ~55-60% failed to reproduce their own (9).

With the increasing number of cases of reproducibility problems and cases where the scientific puzzle can not be fully solved, we should call for measures to increase rigor in the practice of science as has been done. We can assume that the acquisition of scientific knowledge as an incremental process will eventually fill the gaps.  Concomitantly, a radical scientific approach demands that when data says so, we also need to question some of the assumptions on which the practice of science stands.

Embracing Ambiguity

What scientists should do in the light of ambiguity?   First of all, acknowledge it.

By accepting that the one universal truth that science seeks can harbor contrasting perspectives,  scientists should be able to publish contradicting models and results,  without necessarily engaging in conflict or fear of inflicting damage to their own or other researcher’s reputations.

The acknowledgment of ambiguity should take scientists to develop a “YES AND”  attitude as opposed to a “EITHER OR” attitude.  At the laboratory level, in the face of ambiguous results,  scientists should first try to disambiguate them experimentally. However, if efforts are unsuccessful,  groups should consider the possibility of the coexistence of alternative models.  Since our minds operate through linear reasoning, those lines of research should probably be put in the hands of separate lab members or left to be developed by independent groups.  As in the case of light, the knowledge generated through alternative discrepant models could lead to more diversified technological/medical applications than a single model would.

At the scientific community level, recognition of ambiguity is a moral necessity. In the presence of opposing or discrepant reports scientist’s natural tendency is to assume one of them wrong often doubting the skills or integrity of those proposing it.  Instead, ambiguity should be factored in to make the system of professional advancement,  rewards and punishments more just.

While science is no religion, scientists could draw lessons from traditions that see oneness in diversity and opposites.  For instance, says Rabbi Jonathan Sacks, “truth on earth is not nor can aspire to be the whole truth. It is limited, not comprehensive, particular, not universal.  When two propositions conflict it is not necessarily because one is true and the other false. It may be, and often is, that each represents a different perspective on reality, an alternative way of structuring order….In heaven there is Truth,  on earth there are truths. Therefore each culture has something to contribute.  Each person knows something no one else does.” (10).

In the Talmud, the written record of an intergenerational dialog between Rabbis interpreting the Torah to establish laws of observance,  for every opinion sustaining that something should be done in a certain way there are a multiplicity of well-fundamented opinions of why things can not be done in that particular way and should be done differently. Not all opinions make it into the law but all opinions are recorded. In an often-quoted passage (Eruvin 13b), two houses of study, the house of Hillel and the house of Shammai were in constant disagreement.  Whenever the House of  Hillel would propose a certain way of observance, the House of Shamai would propose a  different and more stringent way. The disagreement generated a dispute about the opinion of which house should dictate the law. Ultimately, the story goes,  a Divine voice emerged and proclaimed that both these and those are the words of the living God but the law should be dictated by the house of Hillel.  The reason for following the opinion of the house of Hillel however was not because its opinion was more righteous but because Hillel’s House was more humble, always presenting first the point of view of Shamai before making its own case (11).

The idea of indetermination, ambiguity, and opposites in science is not new, only that the reproducibility crisis is now providing empirical evidence to support it.

Albert Einstein sustained that is the theory that decides what one can be observed.   (12).

Jacques Monod sustained that value-free knowledge is the result not of evidence, but of choice which precedes the collection of evidence and the arrival of performance (13).

Back in 1996, before any reproducibility crisis, Frederick Grinnell called attention to the issue of ambiguity in the practice of science (14). There, among other references, he relates to a passage in Nobel Prize Rita Levi-Montalcini’s autobiography where she refers to Alexander Luria’s “law of disregard of negative information… facts that fit into a preconceived hypothesis attract attention, are singled out, and remembered. Facts that are contrary to it are disregarded, treated as an exception, and forgotten” (15) In this regard Rita Levi-Montalcini makes reference to the tendency to oversee information that could be self-destructive.  No amount of science education can make clear the difference between facts to be remembered and facts to be ignored.  Grinell also refers to the presentation of science as a historically reconstructed, self-consistent, logical process that in the words of Francois Jacob replaces with order the disorder that takes place in the lab (16), and led Nobel Prize  Sir Peter Medawar to write an essay entitled  “Is the scientific paper a fraud ?”(17) where among other things he says “ all scientific work of an experimental or exploratory character starts with some expectation about the outcome of the inquiry…. It is in light of this expectation that some observations are held relevant and others not; that some methods are chosen and others discarded; that some experiments are done rather than others”.

It would be a mistake, however, to assume that everything is ambiguous in science. Some things are, and some things are not.   The paradoxical example of things that are nonambiguous is the genetic code, in which 64 possible codons, a sequence of 3 positions each covered by four possible nucleotides ( A, T, C and G) , code for only  21 amino acids and a signal to stop translation.  The codon possibilities for each amino acid are even larger as the third base is less stringent due to what is known as the wobbling effect.  Still, the engagement preferences of that third base are such that the genetic code is redundant with more than one codon coding for a particular amino acid,  but is non-ambiguous. It could also be argued, of course, that we do not know, during the discovery process,  what pieces of discrepant information may have been consciously or unconsciously ignored in order to arrive to a coherent model.

Rather than escaping ambiguity scientists should embrace it. There is so much to gain from it. From the possibility of a broader menu of technological and medical applications to a more ethical and just way of practicing science and the revelation of a universal framework in which opposites coexist and integrate, opening the path for the confluence of two different approaches to seek truth that have been at odds for long, those of Science and Religion.

References

1)   Jorge Luis Borges, 1949, “The Aleph” in “The Aleph and other stories”.

2)    Jonathan W. Schooler, Metascience could rescue the ‘replication crisis’,  2014, Nature 515, 9

3)    Showalter MR, Hatakeyama J, Cajka T, VanderVorst K, Carraway KL, Fiehn O; Reproducibility Project: Cancer Biology.  Collaborators: Iorns E, Denis A, Perfito N, Errington TM. Replication Study: The common feature of leukemia-associated IDH1 and IDH2 mutations is a neomorphic enzyme activity converting alpha-ketoglutarate to 2-hydroxyglutarate. Elife. 2017;6. pii: e26030. doi: 10.7554/eLife.26030.

4)    Fraser Aird Irawati Kandela Christine Mantis Reproducibility Project: Cancer Biology. Replication Study: BET bromodomain inhibition as a therapeutic strategy to target c-Myc. eLife 2017;6:e21253 DOI: 10.7554/eLife.21253

5)   Horrigan SK, Reproducibility Project: Cancer Biology. 2017a. Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors. eLife 6:e18173.

6)   Horrigan SK, Courville P, Sampey D, Zhou F, Cai S, Reproducibility Project: Cancer Biology. 2017b. Replication Study: Melanoma genome sequencing reveals frequent PREX2 mutations. eLife 6:e21634.

7)   Kandela I, Aird F, Reproducibility Project: Cancer Biology. 2017. Replication Study: Discovery and preclinical validation of drug indications using compendia of public gene expression data. eLife 6:e17044.

8)   Mantis C, Kandela I, Aird F, Reproducibility Project: Cancer Biology. 2017. Replication Study: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs. eLife 6:e17584

9)   Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016 May 26;533(7604):452-4. doi: 10.1038/533452a.

10)  Jonathan Sacks, ed Hava Tirosh-Samuelson and Aaron W. Hughes, 2013,  “Universalizing Particularity.” p53. Brill.

11)The Talmud, (Eruvin 13b)

12) Holton G. Werner Heisenberg and Albert Einstein, 2000, “Creating Copenhagen”.   Symposium  at the Graduate Center of the City University of New York.

13)Jacques Monod, 1970. “Chance and Necessity”

14) Frederick Grinell. Ambiguity in the Practice of Science. 1996. Science. 222(5260) 333.

15)Rita Levi-Montalcini, Transltion Luigi Attardi.  “In Praise of Imperfection.” 1988. (Basic Books, New York), 1988

16)Francois Jacob. F. Philip, Transl. 1987. “ The Statue Within: An Autobiography”, Basic Books, New York.

17) Sir Peter Medawar, Is the scientific Paper a Fraud?  The Listener (12 September 1963), p. 377