Pandemic-Related Submissions: The Challenge of Discerning Signal Amidst Noise

The Editors of Critical Care Medicine The emergence of a novel coronavirus in the final quarter of 2019 precipitated a public health emergency on a global scale. The SARS-CoV-2 virus pandemic, and the spectrum of clinical illnesses labelled COVID-19, has led to more than 6.6. million confirmed cases and more than 390,000 deaths worldwide as of June 5, 2020. More than 1.8 million cases and more than 108,000 attributable deaths have occurred in the United States of America, where this journal is published. This pandemic has also given rise to unprecedented global efforts to describe the pathogen and its mode of spread, the pathobiology of infection and response, and the medical countermeasures that might be undertaken. These include the development of antiviral drugs, antibody treatments aimed at neutralizing the virus, treatments aimed at mitigating the host response, and supportive therapies intended to prevent or attenuate the multisystem complications of the disease. Medical journals, including Critical Care Medicine, have played important roles in reviewing, and ultimately publishing those efforts: since the first report of the pandemic, Critical Care Medicine has received more than 400 original submissions with titles containing either “SARS”, “Coronavirus”, or “COVID”. Many more informal inquiries were either redirected to our sister journal, Critical Care Explorations, or led to our advising the authors not to submit to either journal. The challenges to ourselves, to our readers, and to the world communities have been to discern data from anecdote, information from observation, and ultimately signal amidst noise. Why “Signal” Matters: Evidence and Guidelines These challenges cannot be sidestepped. The practice of evidence-based critical care medicine follows systematic assessment of the available evidence and compilation of guidelines of which strength is based on the quantity and quality of that evidence. Collecting and weighing that evidence requires “thinking slow” to avoid errors in memory, judgement, and decisions. At the same time, the accumulation of case numbers and of deaths galvanizes all of us to “think fast”—to do whatever is required to prevent the next transmission and save the next life. The urgency is reflected in the cover letters that accompany each submission, heartfelt pleas for rapid review and communication. The wisdom of the legendary American lawman and gunfighter, Wyatt Earp, seems apt: “Fast is fine, but accuracy is final. You have to learn to be slow in a hurry.” “Learning to be slow in a hurry” is difficult. The volume of print, electronic, and social media clamoring for our attention is extraordinary. The complexity of the medical publishing world has become even more remarkable: authors of manuscripts submitted to this and other journals now routinely deposit preliminary findings into freely accessible electronic preprint servers. Those authors and their readers subsequently communicate their findings via the channels of social media even before the submission has reached the initial stages of peer-review. The reports are picked up by intelligent sensors—some human, some artificial—and amplified. Unfortunately, some misinformation appears to have been repeated as disinformation, and reliance upon that disinformation may have led to avoidable harms. Early during any emergency, there is a need for reliable accounting of experience. Such rich yet reliable description is important to understand the magnitude of the threat, the nature of the emergency, and the elements that appear to be novel. These preliminary data are important to generate hypotheses that will eventually be tested in the crucible of clinical trials. Early reports equally serve to discourage seemingly logical therapies that lead to unexpected adverse consequences. Even the best efforts to ensure reliable accounting and timely reporting can fail. In one case, an abstract of a peer-reviewed article appearing in a major medical journal describing thousands of COVID-19 patients in New York City had to be urgently revised 48 hours after online publication when it appeared to overstate the mortality of intubated COVID-19 patients. In another case, a widely quoted report in a different major medical journal on the efficacy of masks required retraction when it became apparent that the authors had not accounted for limits of detection in their assay. In yet two more cases, two other high-profile journals retracted a duet of widely-quoted COVID-19 articles because the authors (and those journals) were “unable to validate the primary data sources”. There are other problems with such early accounts, including inadvertent inclusion of (single) patients into multiple studies. While that might at first glance seem to be innocuous, particularly if the studies are reporting on different features of the emergency, confusion follows when submissions first come from single centers, then as reports aggregated from many centers, and finally included into systematic reviews and meta-analyses. (This journal received its first such ‘systematic review and meta-analysis’ submission fewer than 100 days following the first report of the pandemic into the medical literature.) Readers—those at the bedside, and those who create official guidance—may be inappropriately swayed by such inadvertently repeated reporting of individuals. At the same time, there is a clamor for–and need for–clinical guidance. In ordinary times, guideline development begins with integration of basic science information, anecdotal clinical experience, and expert opinion to generate testable hypotheses. Following an iterative sequence of planning, writing, and external review, a clinical trial is designed, approved, funded, and conducted. The results of that clinical trial are then interpreted in the context of prior clinical trials, with special attention to potential limitations of study design and execution. Finally, based on the perceived quality of the evidence, a guideline is produced and assigned a level of confidence. Two problems are immediately evident. First, only a fraction of clinically important questions (for example, the potential repurposing of hundreds of drugs approved for other indications) can be tested rigorously in prospective, randomized, controlled clinical trials. Because preliminary guidance is often based on lower-quality or incomplete evidence, confidence in those guidelines may be limited. Recommendations based on low-quality evidence or expert opinion typically generates skepticism and spirited discussion that predictably culminates in calls for further study. Selecting a clinically important question is thus not only a recursive problem, it becomes a “Catch-22” problem because the number of trials that can be conducted is finite. The other problem, as telegraphed above, is that evidence accumulated early during an emergency is often distorted owing to limited experience and the desire for timely communication. The reason is self-evident: the impetus to save lives by any means possible recalibrates the way patients, scientists, and clinicians perceive truth and make decisions. While preliminary communications that are based on validated information (the number of new cases reported, clinical presentations, unexpected complications) are important, they must be clearly understood to be preliminary. Such preliminary data may offer early insight but must not engender the confidence of larger and broader experience. The Responsibilities of Medical Journals The editors and publishers of professional journals have been pulled into the maelstrom. In the beginning, older data and aging manuscripts “freshened” by adding a paragraph or two asserting (questionable) relevance to the pandemic appeared. These were followed by dozens of case reports and limited series, often complicated by inconsistent care, missing data, and overlapping patients. Experts then attempted to assimilate the experience and translate it into general guidance, often with the imprimatur of global alliances, government agencies, and/or professional societies. Rapid communication, virtual convening of experts, and speedy guideline production led practitioners to expect “living documents” providing “dynamic guidance” on short notice and of limited “shelf life”. The editors of Critical Care Medicine have published some of that guidance. The tension between timeliness and confidence is real: the more specific the guidance, the more likely that it will require revision or even retraction later. This problem is not specific to COVID-19: tight control of glucose and the use of blood transfusions in sepsis (as part of early goal-directed therapy) are two among many examples where rapid and specific guidance had to be “walked back”. That lesson was learned: most COVID-19 guidelines are presented with “boxed warnings” to the effect that the guidance is “interim”, meaning that either the recommendations are based on evidence from related conditions or that the COVID-19–specific data are of uncertain reliability. Our challenge is thus apparent. Medical journals, pressed to serve their readers, publish reliable and actionable information (the “signal”) alongside preliminary, insignificant, and even flawed data (the “noise”) (1). Unfortunately, the distinction between the two may not be apparent to the authors, the reviewers, the editors—nor ultimately to the users. The checks and balances of reflective review were not, and are not, designed to withstand a flood of inchoate data and anecdotes from a variety of sources of varying quality. These challenges may be amplified by strains among the reviewers of the manuscript and the editors of journals, most of whom have competing responsibilities for clinical care and planning among the pandemic. Again, there is a “Catch-22” problem: often the best people to review a manuscript focused on care at the bedside were unable to provide a review because they were properly focused on care at the bedside. The unfortunate outcome is that some published reports–and even some official guidance–will not have benefited from the “normal” systematic processing and scrutiny of information. As hard as we try to avoid contributing, journals can fuel the misinformation problem. The medical journalism response to the emergency has followed a reasonable course. In the current public health emergency–as in so many others–basic research potentially relevant to the emerging disease (e.g., existing information about the biology of coronaviruses) has been resurrected and reviewed for relevance (2,3). Early anecdotal clinical observations regarding the emerging disease have rapidly but unsystematically accumulated (4–12). Drugs that have been tested and used in other clinical settings (e.g., lopinavir-ritonavir) and other compounds with promising preclinical characteristics are rediscovered, re-presented, and promoted in the hope that they will be effective against the new threat (13,14). Agents that have long been approved for one indication (e.g., hydroxychloroquine and famotidine) have been proposed as “off the shelf” weapons to fight the new pathogen (15). There is early reporting that effective vaccines will become available in the future (16) while the antibodies derived from survivors are administered in an attempt to provide a countermeasure (17,18). Existing guidelines for seemingly similar disease states (e.g., the Surviving Sepsis Guidelines) have been revised, updated, and applied (19). Each of these well-meant endeavors is executed with great intention and great intensity with the hope that it will promote understanding, enable treatment, and ultimately help control the pandemic. Under less dire circumstances, such passion might be viewed with skepticism: some of what is rapidly advanced for publication in the name of saving lives will be wrong and patients are harmed. Furthermore, the flood of submissions is so great that we editors will inevitably make our own errors trying to separate signal from noise. That must not stop medical journalism: there is new knowledge to be gained and there are new therapeutic avenues to be evaluated. It was during the 2009–2010 influenza H1N1 pandemic that venovenous extracorporeal membrane oxygenation (VV ECMO) emerged as a key therapy; it is possible that something first tested during this pandemic will enter the critical care armamentarium. We may gain new perspectives into existing concepts of critical care management that need to be replicated (e.g., preliminary experience with the respiratory dysfunction associated with COVID-19 suggests that conventional approaches to management of the acute respiratory distress syndrome [ARDS] may be inappropriate in a subset of patients) (20). Yet distillation of this process takes time. Even where authentic signal can be detected amidst the noise, the journey from clinical observations and expert opinion to guideline development is unlikely to occur with sufficient speed to satisfy the global clamor for evidence-based care. Certain strategies can help accelerate the process. For example, data sharing to hone and test hypotheses and, perhaps more importantly, to detect variation suggesting harm, is essential. At a minimum, common, validated, and verifiable registries will facilitate the emergence of evidence-based best practices while reducing the time from identification to acceptance. The large number of clinical and observational studies rapidly executed lend promise to the idea that we should learn from every patient that we encounter. Best Practices Under Challenging Circumstances Under these challenging circumstances, we believe that editors, authors, and readers assume additional responsibilities. Whatever information is available should be vetted as thoroughly as time constraints permit and then made as widely accessible as possible, as quickly as possible. At the same time, explicit acknowledgment of the limitations of that data must be emphasized and authors may be held to more stringent disclosures of information at onset to avoid republication of data sets from overlapping populations. On-line publication accelerates diffusion of information. With that advantage, however, comes the responsibility to meticulously identify and acknowledge potential failings. We assert that a pandemic imposes an editorial mandate to clearly and publicly acknowledge that emerging data may change validity of what has already been published more rapidly than in usual evolution of science. We have a collective responsibility to update reporting, even when—and especially when—updates negate or reverse findings that were reported previously. Such an action is part of providing “dynamic guidance”. Editors need to remain vigilant and alert our readers to adverse consequences of interventions advocated under our watch. As we encourage and receive signals, we need to do our part to suppress not only the immediate noise but also those echoing aftershocks as noise is perpetuated. Our Response As intermediaries among investigators, reporters, caregivers, and policy-makers, each seeking the imprimatur of responsible peer-review (albeit for different reasons) the roles of editorial leaders and their publications become more significant. Because our journal provides information directly relevant to the care of the very sickest of patients, the editorial leadership of Critical Care Medicine will guide our deliberations and actions according to the following principles when faced with a public health emergency or related crisis. We propose: To modify our editorial review process to balance the need for timely information with the need to exhaustively validate the reported findings. These modifications may include expedited reviews and rendering editorial decisions as soon as sufficient reviewer feedback is received, with less focus on the number of reviewers providing it. We commit to providing rapid decisions that may include referral to our sister journal, Critical Care Explorations, which is explicitly designed to accommodate rapid communication of exploratory (versus definitive) work. We further commit to expedited publication of time-sensitive content. To identify and engage channels where information from multiple, disparate sources are presented. We will responsibly use social media to communicate findings that have passed the peer-review process and are being communicated in the journal. Our journal social media accounts are cautious “custodians of information” not only for regular readers but also for the general public. We must uphold the integrity of the journal when publicizing articles of interest. We acknowledge that threads and narratives in response to our publications constitute extended, if informal, peer review. To require clear distinction of data from interpretation, of interpretation from opinion, and of hypothesis from conclusion. We will require authors and editorialists to illuminate what new knowledge can be reliably taken from the contributions that are accepted for publication. To exclude from publication reports that do not materially contribute new and generalizable insight, with the understanding that novelty is time sensitive and confirmation of key findings may be necessary to confirm generalizability. We will not clutter the literature by publishing reports that do not directly serve our readers in designing, planning, delivering, and evaluating critical care. To evaluate new knowledge as that knowledge accumulates through reviews and syntheses. Those syntheses should be prepared by the subject matter experts who appear best qualified to weigh evidence as it emerges in real time, with the understanding that such syntheses may themselves be exploratory. To promote collegiality and transparency in sharing data among investigators and between scientific publishers to expedite the generation of credible information that can guide the care of those who have been impacted by the emerging threat. The President of the Society of Critical Care Medicine (endorsing): Lewis J. Kaplan The Editors of Critical Care Medicine: Thomas P. Bleck Timothy G. Buchman R. Phillip Dellinger Clifford S. Deutschman John C. Marshall David M. Maslove Henry Masur Margaret M. Parker Donald S. Prough Aarti Sarwal Jonathan E. Sevransky Jean-Louis Vincent Jerry J. Zimmerman