The Reproducibility Wars: Successful, Unsuccessful, Uninterpretable, Exact, Conceptual, Triangulated, Contested Replication

The recent publication of first results from the “Reproducibility Project: Cancer Biology” has stirred debate. The project synopsis put together by Nosek and Errington (1) tried to describe carefully what replication means, how to judge whether “same” (or different) results emerge in a replication experiment, and how to interpret divergent results in original vs reproducibility studies. However, multiple other commentators on these first reproducibility studies have enhanced our uncertainty. Almost every commentator reached a somewhat different conclusion. Reproducibility of inferences has been dismal. Several authors of the original papers along with other commentators have questioned the reproducibility effort. These retorts typically defend the original findings, interpreting the replications as more successful than unsuccessful. They question replications on multiple fronts, e.g., inappropriate statistical methods or poor experimental competence. They lament the inappropriate shaming consequences, when poorly done replication efforts tarnish great scientists. They worry that reproducibility checks destroy discovery and stall efforts to translate promising research. They wonder whether we should waste money on replication. The reproducibility wars are not exclusive to laboratory science. Last year, a similar debate erupted with a Technical Comment exchange on the respective reproducibility project on psychology (2), although the available data were far more extensive (100 experiments instead of just 5 preliminary ones) and had a massive participation of top psychologists (270 scientists and their teams) trying to reproduce results. Some famous psychology academics nevertheless concluded that their field had no reproducibility problem and reproducibility was misleading business. This position is immediately suspect. If psychological science is so perfect, how can it be that 270 of the best psychologists in the world working under optimal conditions of openness and most rigorous protocols and methods could get everything so badly wrong? If the most closely controlled and scrutinized corpus of experiments ever done on psychological science …