X-WAS, Traditional Epidemiology, and Policy Action

We thank Daniele Fallin and Linda Kao,1 George Davey Smith,2 and Shalom Wacholder3 for their insightful commentaries. Fallin and Kao1 doubt whether X-WAS is the future of all epidemiology and whether the rest of epidemiology beyond genomics is as biased as we implied.4 Epidemiology, and observational research in general, comprises a vast cluster of diverse disciplines; some areas may be less biased than human genome epidemiology in the candidate gene era, but others are probably even more biased. There are hundreds of possible biases.5 No one denies that traditional epidemiology and observational studies have been successful at times; we can add numerous examples of successful identification of risk factors6 besides the 2 mentioned by Fallin and Kao.1 However, as indicated by fellow commentator, Davey Smith,2 the average track record of observational epidemiology to date has been poor, almost a systemic failure.7 The successes are needles in a haystack of associations reported in thousands and possibly millions of studies. A quick PubMed search with [risk factor OR (case-control OR cohort OR cross-sectional) OR association] yields 1,489,912 articles. Perusing but a small sample, we find 90% of these retrievals are observational epidemiologic studies, while many other observational studies are probably not captured by this preliminary search. Hundreds of thousands of scientists have been involved in observational studies across diverse domains. Some of them have not even realized that this is the approach they have been using, and that they may have employed spurious inferential methods. Some prolific cohorts have published well over 1000 papers, often each examining 1 or 2 exposures for one disease outcome. We do not think that such fragmentation serves our science well. Fallin and Kao indicate that findings from GWAS studies hold translational potential but are concerned that stringent criteria will limit the pursuit of important paths to translation. Successful translation occurs sparingly in biomedical science.8 This difficulty should be acknowledged and communicated to the wider public. We do not expect massive, agnostic approaches to make the translation process much easier. However, a rigorous approach with appropriate FP:FN stringency, large sample sizes, transparent design, and nonselective reporting will ensure a starting point with fewer false leads. False-positives continue to haunt the literature, even after being roundly refuted.9 Moreover, putative discoveries from observational studies continue to have a high refutation rate even when findings have reached the point of being close to translation.10 The current emphasis on translation by funders may exacerbate the false positive problem, as it presupposes validity of research findings. Funding incentives to rush to the bedside with presumed treatments will likely encourage bad science. We agree with the principle of including additional lines of evidence, such as animal models,11 in vitro work, and other sources of biologic plausibility. However, biologic plausibility also needs an equally systematic and transparent approach to the evidence—otherwise scientists may selectively invoke those pieces of the evidence that best match their results or prejudices.12 Biases related to statistical testing and selective reporting are also highly prevalent in these additional lines of biologic evidence.13–16 Wacholder3 nicely summarizes some differences between human genome epidemiology and other disciplines. He points out that, with more stringent FP:FN criteria, the power of current studies would be eroded. However, if one was able to put together all the fragmented single-study efforts in traditional epidemiology, the cumulative sample sizes of the resulting consortia would often be as large as or larger than the typical samples amassed in current genetic consortia. Future efforts may be even more far-reaching. If 50 biobanks are launched around the world, with an average of 500,000 participants each, the total sample size is 25,000,000 participants. Some nationwide studies can already accommodate information from many millions of participants each. Unless criteria of discovery and validation are stringent, these studies may show nominally statistically significant results for almost any tested association. Wacholder also highlights differences in traditional versus genome epidemiology in cost, free availability of measurements, and use of combined (pooled) analyses. We do not see why traditional epidemiology cannot follow the paradigm of genetics in these aspects. The cost of genetic measurements has decreased a million-fold in the last 20 years. One can also envision a sharp decrease in the cost of nongenetic measurements and data gathering. Cost is high when technologies are home-made, idiosyncratic, and nongeneralizable. Conversely, cost routinely decreases when a technology becomes standardized and then applied by many scientists using common analytic plans and converging purposes. Several envirome measurement platforms are already available and continuously being improved—including tools to capture the epigenome, metabolome, and microbiome; administrative databases and national registries; and electronic epidemiology, to name a few.17 We also see no rationale why traditional epidemiologic data should not be freely available,18 and why prospectively designed combined analyses in overarching consortia should not be the norm. Once these deficits have been amended, one can deal more seriously with the other differences that may indeed be more specific to nongenetic epidemiology, including the range of prior odds and the highly dense correlation pattern.19,20 Davey Smith2 shows how Mendelian randomization bridges the gap between genetic and nongenetic epidemiology and offers a wealth of new possibilities. We believe that it is important to bring different platforms of measurement to coexist within the same cohorts and biobanks.20 To date, most studies that have extensive genomic measurements have limited or no information on the envirome, and studies that are strong in environmental measurements often lack interest in genomics. Hopefully, this unfortunate dissociation will be gradually remedied. Finally, what does this all mean for policy and decision-making? As we pointed out originally4 and as our commentators further elaborated, scientific appraisal of the evidence is not the same as making policy decisions based on this evidence. We simply suggest that scientists should give transparent estimates about the FP:FN ratio of the field in which the information was obtained, systematic and transparent synopses of the prior evidence in the field, and estimates about the posterior odds that take the range of existing uncertainty into account. Scientific studies, when correctly executed and validated, inform us about what is, but not what ought to be. This may seem obvious to some, but one need only follow the newspapers or TV news to see what happens when scientists become advocates based on their own research findings.