An umbrella review of effect size, bias, and power across meta‐analyses in emergency medicine

Abstract Objectives The objective of this study was to conduct an umbrella review of therapeutic studies relevant to emergency medicine, analyzing patterns in effect size, power, and signals of potential bias across an entire field of clinical research. Methods We combined topic‐ and journal‐driven searches of PubMed and Google Scholar for published articles of systematic reviews and meta‐analyses (SRMA) relevant to emergency medicine (last search in November 2020). Data were screened and extracted by six investigators. Redundant meta‐analyses were removed. Whenever possible for each comparison we extracted one meta‐analysis on mortality with the most events and one meta‐analysis on a nonmortality outcome with the most studies. From each meta‐analysis we extracted all individual study effects; outcomes were converted to odds ratios (ORs) and placed on a common scale where an OR < 1.0 represents a reduction in a harmful outcome with an experimental treatment versus control. Outcomes were analyzed at the level of individual studies and at the level of summary effects across meta‐analyses. Results A total of 332 articles contained 431 eligible meta‐analyses with a total of 3,129 individual study outcomes; of these, 2,593 (83%) were from randomized controlled trials. The median OR across all studies was 0.70. Within each meta‐analysis, the earliest study effect on average demonstrated larger benefit compared to the overall summary effect. Only 57 of 431 meta‐analyses (13%) both favored the experimental intervention and did not show any signal of small study effects or excess significance, and of those only 12 had at least one study with 80% or higher power to detect an OR of 0.70. Of these, no interventions significantly decreased mortality in well‐powered trials. Although the power of studies increased somewhat over time, the majority of studies were underpowered. Conclusions Few interventions studied within SRMAs relevant to emergency medicine seem to have strong and unbiased evidence for improving outcomes. The field would benefit from more optimally powered trials.