Publications

Findings from clinical and biological studies are often not reproducible when tested in independent cohorts. Due to the testing of a large number of hypotheses and relatively small sample sizes, results from whole-genome expression studies in particular are often not reproducible. Compared to single-study analysis, gene expression meta-analysis can improve reproducibility by integrating data from multiple studies. However, there are multiple choices in designing and carrying out a meta-analysis. Yet, clear guidelines on best practices are scarce. Here, we hypothesized that studying subsets of very large meta-analyses would allow for systematic identification of best practices to improve reproducibility. We therefore constructed three very large gene expression meta-analyses from clinical samples, and then examined meta-analyses of subsets of the datasets (all combinations of datasets with up to N/2 samples and K/2 datasets) compared to a 'silver standard' of differentially expressed genes found in the entire cohort. We tested three random-effects meta-analysis models using this procedure. We showed relatively greater reproducibility with more-stringent effect size thresholds with relaxed significance thresholds; relatively lower reproducibility when imposing extraneous constraints on residual heterogeneity; and an underestimation of actual false positive rate by Benjamini-Hochberg correction. In addition, multivariate regression showed that the accuracy of a meta-analysis increased significantly with more included datasets even when controlling for sample size.

Abstract Objectives To examine the prevalence of comparisons of surgery to drug regimens, the strength of evidence of such comparisons, and whether surgery or the drug intervention was favored. Design Systematic review of systematic reviews (umbrella review) Data sources Cochrane Database of Systematic Reviews (CDSR) Eligibility criteria and synthesis of results Using the search term “surg*” in CDSR, we retrieved systematic reviews of surgical interventions. Abstracts were subsequently screened to find systematic reviews that aimed to compare surgical to drug interventions; and then, among them, those that included any randomized controlled trials (RCTs) for such comparisons. Trial results data were extracted manually and synthesized into random-effects meta-analyses. Results Overall, 188 systematic reviews intended to compare surgery versus drugs. Only 41 included data from at least one RCT (total, 165 RCTs with data) and covered a total of 103 different outcomes of various comparisons of surgery versus drugs. A GRADE assessment was performed by the Cochrane reviewers for 87 (83%) outcomes in the reviews, indicating the strength of evidence was high in 4 outcomes (4%), moderate in 22 (21%), low in 27 (26%) and very low in 33 (32%). Based on 95% confidence intervals, the surgical intervention was favored in 38/103 (37%), and the drugs were favored in 13/103 (13%) outcomes. Of the outcomes with high GRADE rating, only one showed conclusive superiority (sphincterotomy was better than medical therapy for anal fissure). Of the 22 outcomes with moderate GRADE rating, 6 (27%) were inconclusive, 14 (64%) were in favor of surgery, and 2 (9%) were in favor of drugs. Conclusions Though the relative merits of surgical versus drug interventions are important to know for many diseases, high strength randomized evidence is rare. More randomized trials comparing surgery to drug interventions are needed. Protocol registration https://osf.io/p9x3j Funding The work of John Ioannidis has been funded by an unrestricted gift from Sue and Bob O’Donnell. Anaïs Rameau is supported by a Paul B. Beeson Emerging Leaders Career Development Award in Aging (K76 AG079040) from the National Institute on Aging and by the Bridge2AI award (OT2 OD032720) from the NIH Common Fund. Anirudh Saraswathula is supported by the National Institute on Deafness and Other Communication Disorders training grant 2T32DC000027. Financial disclosure Anaïs Rameau is a medical advisor for Perceptron Health, Inc. Summary boxes Section 1: What is already known on this topic Many conditions and diseases can be managed either with surgery or with drugs. Comparative effectiveness of different treatment options is important to know for shared decision-making. Section 2: What this study adds An assessment of the entire Cochrane Database of Systematic Reviews found 188 reviews that intended to assess surgical versus drug interventions, but only 41 had at least one randomized trial. Only four of the 103 assessed outcomes had high strength of evidence according to GRADE assessments. More evidence is needed to compare the relative merits of surgical and drug interventions and sequestration of these major modes of interventions should be overcome in clinical trial agendas.

Abstract Background: Among different investigators studying the same exposures and outcomes, there may be a lack of consensus about potential confounders that should be considered as matching, adjustment, or stratification variables in observational studies. Concerns have been raised that confounding factors may affect the results obtained for the alcohol-ischemic heart disease relationship, as well as their consistency and reproducibility across different studies. Therefore, we assessed how confounders are defined, operationalized, and discussed across individual studies evaluating alcohol and ischemic heart disease risk. Methods: For observational studies included in a recent alcohol-ischemic heart disease meta-analysis, we identified all variables adjusted, matched, or stratified for in the largest reported multivariate model (i.e. potential confounders). We recorded how the variables were measured and grouped them into higher-level confounder domains. Abstracts and Discussion sections were then assessed to determine whether authors considered confounding when interpreting their study findings. Results: 85 of 87 (97.7%) studies reported multivariate analyses for an alcohol-ischemic heart disease relationship. The most common higher-level confounder domains included were smoking (79, 92.9%), age (74, 87.1%), and BMI, height, and/or weight (57, 67.1%). However, no two models adjusted, matched, or stratified for the same higher-level confounder domains. Most (74/87, 85.1%) articles mentioned or alluded to “confounding” in their Abstract or Discussion sections, but only one stated that their main findings were likely to be affected by residual confounding. There were five (5/87, 5.7%) authors that explicitly asked for caution when interpreting results. Conclusion: There is large variation in the confounders considered across observational studies evaluating alcohol and ischemic heart disease risk and almost all studies spuriously ignore or eventually dismiss confounding in their conclusions. Given that study results and interpretations may be affected by the mix of potential confounders included within multivariate models, efforts are necessary to standardize approaches for selecting and accounting for confounders in observational studies.

Journals exert considerable control over letters, commentaries, and online comments that criticize prior research (post-publication critique). We assessed policies (Study One) and practice (Study Two) related to post-publication critique at 15 top-ranked journals in each of 22 scientific disciplines (N=330 journals). 207 (63%) journals accepted post-publication critique and often imposed limits on length (median 1000, interquartile range [IQR] 500-1200 words) and time-to-submit (median 12, IQR 4-26 weeks). The most restrictive limits were 175 words and 2 weeks; some policies imposed no limits. Of 2066 randomly sampled research articles published in 2018 by journals accepting post-publication critique, 39 (1.9%, 95% confidence interval [1.4, 2.6]) were linked to at least one post-publication critique (there were 58 post-publication critiques in total). Of the 58 post-publication critiques, 44 received an author reply, of which 41 asserted that original conclusions were unchanged. Clinical Medicine had the most active culture of post-publication critique: all journals accepted post-publication critique and published the most post-publication critique overall, but also imposed the strictest limits on length (median 400, IQR 400-550 words) and time-to-submit (median 4, IQR 4-6 weeks). Our findings suggest that top-ranked academic journals often pose serious barriers to the cultivation, documentation, and dissemination of post-publication critique.

Displaying the real-time behavior of critical signals on VLSI chips is difficult and can require expensive test equipment. We present a simple sampling technique to display the analog waveforms of high bandwidth on-chip signals on a laboratory oscilloscope. It is based on the subsampling of periodic signals. This circuit was used to verify the operation of a recent low-power SRAM design.