Author response: Evaluating mesenchymal stem cell therapy for sepsis with preclinical meta-analyses prior to initiating a first-in-human trial

Article Figures and data Abstract eLife digest Introduction Results Discussion Materials and methods Appendix 1. Systematic search strategies References Decision letter Author response Article and author information Metrics Abstract Evaluation of preclinical evidence prior to initiating early-phase clinical studies has typically been performed by selecting individual studies in a non-systematic process that may introduce bias. Thus, in preparation for a first-in-human trial of mesenchymal stromal cells (MSCs) for septic shock, we applied systematic review methodology to evaluate all published preclinical evidence. We identified 20 controlled comparison experiments (980 animals from 18 publications) of in vivo sepsis models. Meta-analysis demonstrated that MSC treatment of preclinical sepsis significantly reduced mortality over a range of experimental conditions (odds ratio 0.27, 95% confidence interval 0.18–0.40, latest timepoint reported for each study). Risk of bias was unclear as few studies described elements such as randomization and no studies included an appropriately calculated sample size. Moreover, the presence of publication bias resulted in a ~30% overestimate of effect and threats to validity limit the strength of our conclusions. This novel prospective application of systematic review methodology serves as a template to evaluate preclinical evidence prior to initiating first-in-human clinical studies. https://doi.org/10.7554/eLife.17850.001 eLife digest Most attempts to transform exciting findings from laboratories into clinical treatments are unsuccessful. One reason for this may be the failure to consider all of the laboratory work that has been performed before deciding to test a treatment on patients for the first time. In particular, negative findings (that suggest that a potential new treatment is ineffective) may be overlooked. Stem cells may help to treat life-threatening infections, but this has not been tested in human patients. However, the effectiveness of stem cell treatments has been tested in animals that act as models of human infection. Before deciding to begin a clinical trial of stem cell therapy for life-threatening infections, Lalu et al. performed an exhaustive search to find all the studies in which stem cells were used to treat animal models of infection. Combining the results of all of these studies using particular analysis techniques revealed that stem cell therapy increased the survival of these animals overall. These positive effects were seen over a range of different experimental conditions (for example, when treating the animals with different doses of stem cells, or giving the doses at different times). Lalu et al. also identified some limitations with most of the laboratory studies that had tested stem cell therapy for infections. Many of the studies used animal models that may not be the best representations of humans with severe infection. In addition, many of the scientists did not report that they had used methods (such as randomization) that would generate the most confidence in their results. Despite these limitations, there was a lot of consistency in the reported results. Overall, the results support the decision to proceed to a clinical trial that tests the effectiveness of stem cells for treating human infections. More generally, Lalu et al.'s analysis demonstrates a way of considering all laboratory evidence before deciding to proceed to a first clinical trial in humans. This may help researchers to identify promising therapies to further develop, and also to identify potential failures before they are tested in patients. https://doi.org/10.7554/eLife.17850.002 Introduction The decision to initiate an early phase clinical trial requires careful evaluation of the benefits and risks of a novel intervention. However, for first-in-human studies for which there is no prior clinical experience, the assessment of potential therapeutic efficacy must rely solely on the preclinical investigations. Although regulatory guidance exists for the conduct of preclinical evaluation of novel therapies (U.S. Department of Health and Human services, 2013), there is little guidance to help stakeholders summarize and assess the benefit and risks of novel therapies prior to first-in-human studies. As a result, the evidence from individual preclinical studies is often summarized and described in a non-systematic and potentially biased manner (Food and Drug Administration, 2015). Here, we present an approach to transparently evaluate preclinical evidence of a therapy prior to its potential clinical translation. Our exemplar is mesenchymal stem cell (MSC) therapy for sepsis. A selective narrative summary of preclinical evidence has significant limitations because the methods used to identify studies are neither comprehensive nor transparent (Sena et al., 2014). This is of particular concern given that studies replicating high profile experiments fail in up to 50–90% of attempts (Begley and Ellis, 2012; Scott et al., 2008; Steward et al., 2012) and significant publication bias results in a skewed representation of effects (Sena et al., 2010). Further, fewer than 5% of high impact preclinical reports are clinically translated (Contopoulos-Ioannidis et al., 2003) and only 11% of clinically tested agents receive licensing (Kola and Landis, 2004). Thus trialists have based predictions of clinical success of novel therapies on flawed data and an inappropriately highly selected and positive preclinical evidence base (Grankvist and Kimmelman, 2016). Systematic reviews and meta-analyses have become very popular because they can overcome many of these challenges by promoting the transparent evaluation of therapies. Systematic reviews are guided by a protocol with explicit methods to identify, synthesize (which may include meta-analysis), and appraise all investigations pertinent to a particular research question. Similarly, meta-analysis enables pooling of effect sizes across studies and increases statistical power by reducing standard error around the average effect size, providing a more precise estimate of an overall treatment effect (Sena et al., 2014; Cohn and Becker, 2003). Systematic reviews and meta-analyses have long been regarded as essential tools to summarize and evaluate clinical research (Higgins and Green, 2009) and have become a requisite component of grant applications for clinical trials (Canadian Institutes of Health Research, 2016); however, the application of these tools to preclinical studies has been limited. Preclinical systematic reviews may help predict the magnitude and direction of novel therapeutic effects in high stakes first-in-human trials. For example, preclinical systematic reviews of stroke (Horn et al., 2001) and heart failure (Lee et al., 2003) therapies demonstrated that the resulting negative clinical trials could have been predicted had available preclinical evidence been analyzed in a rigorous manner. Thus, thousands of patients may have avoided exposure to potential risk without any benefit (Kalra et al., 2002; Shuaib et al., 2007). Similarly, previous preclinical systematic reviews have demonstrated that failure to report threats to methodological quality (i.e. internal validity, risk of bias) and construct validity (i.e. extent a model corresponds to the human condition it is intended to represent [Henderson et al., 2013]) influence treatment effect sizes (Crossley et al., 2008; Hirst et al., 2014; Macleod et al., 2008, 2015; Rooke et al., 2011). Unlike this 'retrospective' approach that has been described in previous studies, a prospective application of preclinical systematic review methodology may help delineate the limits of a therapy prior to first-in-human application. Our preclinical systematic review was conducted prior to the initiation of a Phase 1/2 clinical trial of immunomodulatory cell therapy (mesenchymal stromal cells, mesenchymal stem cells [MSCs], "adult stem cells") for septic shock (NCT02421484). The specific question addressed was: In preclinical in-vivo animal models of sepsis, what is the effect of MSC administration (compared to control treatment) on death? Septic shock is the result of an overwhelming systemic infection; it is one of the most common and acutely devastating health problems in the intensive care unit with a 90-day mortality rate of approximately 20–30% despite modern therapy (Peake et al., 2014; Mouncey et al., 2015; Stevenson et al., 2014). It is caused by a maladaptive mismatch between host inflammatory response and pathogenic stimuli which leads to organ failure and death. MSCs are ubiquitous cells (da Silva Meirelles et al., 2006) that support tissue repair and are mobilized under inflammatory conditions (Hannoush et al., 2011; Rochefort et al., 2006). Exogenously administered MSCs represent an especially attractive therapeutic for sepsis because they have antibacterial and organ protective effects, in addition to their immune modulatory functions (Walter et al., 2014). We quantitatively summarized the results of all preclinical studies of MSC therapy for in vivo animal models of sepsis to predict effect size and establish an ethical basis for exposing high-risk patients to this novel therapy. This is the first systematic evaluation of a novel biologic therapy prior to initiating a first-in-human trial. We believe our approach serves as a roadmap to transparently evaluate a preclinical therapy prior to its potential clinical translation. This study has been written in an explicatory manner so that other preclinical and translational researchers not familiar with systematic review methodology may replicate our approach. Readers wishing to replicate our approach for their research agendas are directed to the methods section where explanations are provided in greater depth, and encouraged to contact the authors for further guidance. Results Search results and study characteristics Our systematic search of MEDLINE, Embase, BIOSIS, and Web of Science yielded 3114 records. Following deduplication and screening, 18 studies were included in the review (Figure 1). These studies were published over a six year period (2009 to 2015) and corresponded to 20 unique experiments and involved a total of 980 animals (Table 1) (Bi et al., 2010; Chang et al., 2012; Chao et al., 2014; Gonzalez-Rey et al., 2009; Hall et al., 2013; Kim et al., 2014; Krasnodembskaya et al., 2012; Li et al., 2012; Liang et al., 2011; Luo et al., 2014; Mei et al., 2010; Nemeth et al., 2009; Pedrazza et al., 2014; Sepúlveda et al., 2014; Yang et al., 2015; Zhao et al., 2013, 2014; Zhou et al., 2014). Six authors were contacted for additional information and all replied. Figure 1 Download asset Open asset Preferred reporting items for systematic reviews and meta-analysis (PRISMA) flow diagram for study selection. https://doi.org/10.7554/eLife.17850.003 Table 1 General characteristics of preclinical studies investigating the efficacy of mesenchymal stromal cells in models of sepsis. https://doi.org/10.7554/eLife.17850.004 Author year CountrySpecies, Strain, GenderSepsis modelResuscitationMSC source, CompatibilityMSC DoseTime (hours)*MSC routeControlGonzalez-Rey et al. (2009)A SpainMouse BALB/c, NRCLP (1 × 22 G)NoneAdipose Xenogenic or Allogeneic1.0 × 1064IPDMEMGonzalez-Rey et al. (2009)B SpainMouse BALB/c, NRLPS (i.p.)NoneAdipose Xenogenic1.0 x 106 or3.0 x 1050.5IPDMEMNemeth et al. (2009) United StatesMouse C57BL/6, MCLP (2 × 21 G)Fluid and antibioticsBone marrow Allogeneic1.0 × 1060 or 1IVPBS or FibroblastBi et al. (2010) ChinaMouse C57BL/6, NRCLP (2 × 21 G)NoneBone marrow Xenogenic1.0 × 1061 1IVPBSMei et al. (2010)A CanadaMouse C57BL/6J, FCLP (1 × 22 G)FluidBone marrow Syngeneic2.5 × 1056IVNSMei et al. (2010)B CanadaMouse C57BL/6J, FCLP (1 × 18 G)Fluid and antibioticsBone marrow Syngeneic2.5 × 1056IVNSLiang et al. (2011) ChinaRat Wistar, FLPS (i.v.)NoneBone marrow Syngeneic1.0 × 1062IVNSChang et al. (2012) ChinaRat SPD, MCLP (2 × 18 G)NoneAdipose Autologous3 × 1.2 × 1060.5, 6 then 18IPNSKrasnodembskaya et al. (2012), USAMouse C57BL/6J, MP. aeruginosa (i.p.)NoneBone marrow Xenogenic1.0 × 1061IVPBS FibroblastLi et al. (2012) ChinaRat SPD, MLPS (i.p.)NoneUmbilical cord Xenogenic5.0 × 1051IVNS or FibroblastHall et al. (2013) USAMouse BALB/c, MCLP (2 × 21 G)NoneBone marrow Syngeneic1 × 5.0× 105 + 2× 2.5 × 1052 then 24 then 48IVPBS or FibroblastZhao et al. (2013) ChinaRat SPD, FLPS (i.v.)NoneBone marrow Syngeneic2.5 ×1062IVNSChao et al. (2014) TaiwanRat Wistar, MCLP (1 × 18 G)NoneBone Marrow or Umbilical Cord Xenogenic5.0 × 1064IVPBSKim et al. (2014) CanadaMouse C57BL/6, MSEB+ (i.p)NoneBone marrow Syngeneic2.5 × 1053IVPBSLuo et al. (2014) ChinaMouse C57Bl/6, MCLP (2 × 21 G)FluidBone marrow Syngeneic1.0 × 1063IVNSPedrazza et al. (2014) BrazilMouse C57BL/6, ME. coli (i.p.)NoneAdipose Syngeneic1.0 × 1060IVPBSSepulveda et al. (2014) SpainMouse BALB/c, MLPS (i.p.)NoneBone Marrow Xenogenic1.0 × 1060.5IPPBSZhao et al. (2014) ChinaMouse C57BL/6, MCLP (NR)NoneUmbilical cord Xenogenic1.0 × 1061IVNSZhou et al. (2014) ChinaMouse NOD SCID, MLPS+ (i.p.)NoneUmbilical Cord Xenogenic2.0 × 1066IVNo treatmentYang et al. (2015) ChinaMouse NOD SCID, MLPS+ (i.p.)NoneUmbilical cord Xenogenic5.0 × 1050IVDMEM Legend: * = Time of delivery post-sepsis induction, + = Models also administered D-galactosamine, CLP = Cecal ligation and puncture, DMEM = Dulbecco's modified Eagle's medium, i.p. = Intraperitoneal, i.v. = Intravenous, LPS = Lipopolysaccharide, NR = Not reported, NOD SCID = NOD.Cg-Prkdcscid Il2rgtm1Wjl/SzJ (immunodeficient), NS = Normal saline, PBS = Phosphate buffered saline, SEB = Staphylococcal enterotoxin B, SPD = Sprague-Dawley. All experiments used rodents, and most were mice (80%). Several methods were used to establish sepsis or sepsis-like pathophysiology, including cecal-ligation and puncture (50%), live bacterial injection (10%), and bacterial component injection (40%). Tissue sources of MSCs included bone marrow (60%), adipose tissue (20%), and umbilical cord (20%). Similarly, immunological compatibility between donor MSCs and recipients varied between xenogenic (50%), syngeneic (40%), allogeneic (5%) and autologous (5%). Two of ten experiments with xenogenic cells used immunocompromised mice, while the remainder used immunocompetent mice. Total doses of MSCs ranged from 2.5 × 105 to 5.0 × 106 and most studies administered cells as a single dose (90%) either intravenously (80%) or intraperitoneally (20%). MSC therapy was initiated between 0 to 6 hr after experimental induction of the disease state. Effect of MSCs on sepsis mortality in rodents MSC therapy in preclinical models of sepsis significantly reduced the overall odds of death (odds ratio (OR) 0.27, 95% confidence interval (CI) 0.18–0.40 (Figure 2). Since it is important to consider the consistency of results between studies, we calculated the I2 test, which demonstrated a low degree of heterogeneity across studies (I2 = 33%). The reduction in mortality was maintained regardless of when death occurred, whether considering deaths before two days after induction of sepsis (OR 0.31, 95% CI 0.21–0.46), between two and four days (OR 0.20, 95% CI 0.11–0.38), or more than four days (OR 0.18, 95% CI 0.11–0.32) (Figure 3). Figure 2 with 11 supplements see all Download asset Open asset Forest plot summarizing effects of mesenchymal stromal cell (MSC) therapy on mortality in preclinical models of sepsis and endotoxemia. Point estimates (odds ratio) and 95% confidence intervals (CI) are depicted for individual studies; size of point estimate depicts relative contribution to pooled effect. A pooled meta-analytic summary (random effects model) of overall effect of MSC therapy on mortality is depicted by the diamond at the bottom of the plot (vertical points represent odds ratio point estimate and horizontal points represent 95% CIs). Heterogeneity is represented with the I2 statistic. Data from Pedrazza et al. (2014) was included in total counts but not included in meta-analysis due to 100% mortality in both study arms. https://doi.org/10.7554/eLife.17850.005 Figure 3 Download asset Open asset Forest plot summarizing relationship of mesenchymal stromal cell (MSC) therapy on mortality over time in preclinical models of sepsis and endotoxemia (outcome windows: ≤2 days, >2 to ≤ 4 days, > 4 days). Point estimates (odds ratio) and 95% confidence intervals (CI) are depicted for individual studies; size of point estimate depicts relative contribution to pooled effect. A pooled meta-analytic summary (random effects model) of overall effect of MSC therapy on mortality is depicted by the diamond at the bottom of each time interval (vertical points represent odds ratio point estimate and horizontal points represent 95% CIs). Heterogeneity is represented with the I2 statistic. Data from Pedrazza et al. (2014) was included in total counts but not included in meta-analysis due to 100% mortality in both study arms. https://doi.org/10.7554/eLife.17850.017 Assessment of threats to external validity/generalizability The effects of therapies may not be sustained under varied experimental conditions, so we evaluated the generalizability and replicability of results by analyzing efficacy in pre-specified sub-groups. Heterogeneity (i.e. I2 statistic) was low to moderate unless otherwise stated. Similar efficacy was noted regardless of the compatibility of donor MSCs with recipient animal (syngeneic vs. allogeneic vs. xenogenic, Figure 2—figure supplement 1), dose of MSC (<1.0 × 106 cells vs. ≥1.0 × 106 cells, Figure 2—figure supplement 2), and timing of a single dose of MSCs (less than or equal to 1 hr vs. 1–6 hr after disease induction, Figure 2—figure supplement 3). Intravenous administration of MSCs demonstrated efficacy (OR 0.28, 95% CI 0.20–0.40); whereas intraperitoneal administration of MSCs did not have a statistically significant effect (OR 0.21, 95% CI 0.02–1.89; Figure 2—figure supplement 4) and had high heterogeneity (I2 = 78%), suggesting a high degree of inter-study variability. Significant effects were seen using MSCs derived from bone marrow (OR 0.13, 95% CI 0.05–0.35) and umbilical cord (OR 0.30, 95% CI 0.21–0.43; Figure 2—figure supplement 5), but the MSCs derived from adipose tissue did not demonstrate statistically significant efficacy (OR 0.35, 95% CI 0.03–4.39, I2 = 79%). Two studies administered multiple doses of MSCs, with one demonstrating benefit and the other having no statistically significant effect. The multiple dose study with no effect was also the only investigation of autologous cells (Chang et al., 2012). MSCs administered to mice were effective (OR 0.23, 95% CI 0.15–0.36) however MSC administration to rats did not produce a statistically significant effect (OR 0.47, 95% CI 0.18–1.21; Figure 2—figure supplement 6). Neither the sex of the diseased animal nor the model used (cecal ligation and puncture vs. live bacterial injection vs. lipopolysaccharide or other bacterial product) influenced efficacy (Figure 2—figure supplements 7 and 8). The addition of resuscitation (fluids or antibiotics, which are current clinical standards of therapy) did not influence the protective effect of MSCs (Figure 2—figure supplement 9). The comparator control group (phosphate buffered saline vs. fibroblast vs. normal saline vs. medium) had no effect; but, the one study that did not administer vehicle to the control animals did not demonstrate a statistically significant effect of MSC therapy (Zhou et al., 2014) (Figure 2—figure supplement 10). Assessment of threats to internal validity (methodological quality/risk of bias) Practices such as blinding and randomization are known to affect the magnitude of effect in both clinical and preclinical studies. To determine if these threats to internal validity influenced our findings, we evaluated the risk of bias of included studies (Table 2). None of the experiments were considered low risk of bias across all six domains of methodological quality. Forty-five percent of experiments reported that the animals were randomized, none described methods of sequence generation or how allocation concealment was achieved. Similarly, no studies described blinding of personnel performing the experiments. One study did not blind assessors for the outcome of mortality, which may be of concern given that surrogate endpoints (i.e. not true death due to animal welfare concerns) were assessed (Kim et al., 2014); the remaining studies were assessed as 'unclear' as insufficient details of outcome assessment were reported. An assessment of high risk of bias for incomplete outcome data occurred in 10% of studies as reported from methods to in of experiments the were not in both the methods and results in to studies reported an for of study sample size included a calculated sample size, Table 3). the of studies that and reported internal validity an analysis to determine the effects of high vs. low risk of bias on the effect size was not Table 2 Risk of bias assessment of preclinical studies investigating the efficacy of mesenchymal stromal cells in models of sepsis. of of outcome outcome outcome et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. Legend: = risk of = risk of = risk of bias of Assessment for risk = assessors were to the study when mortality surrogate endpoints or animals were to = information to determine if outcome assessors were assessment or if animals were to Risk = assessors not to the study and death was to surrogate risk = were between methods and results for the mortality = was either not in the methods or in the and there is insufficient information to risk = were not between methods and results for the mortality risk = The methods section mortality as a pre-specified outcome risk = The mortality outcome was in the results but not pre-specified in the methods Table 3 Risk of bias assessment of preclinical studies investigating the efficacy of mesenchymal stromal cells in models of sepsis. of of size et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. et al. Legend: * = modified from risk of bias = risk of = risk of = risk of bias risk = of disease equal between experimental = of disease risk = of disease between experimental risk = were an animal = risk = in in animal risk of bias was assessed to of of and pre-specified sample size of risk = of no = was not reported. risk = was by of risk = reported on no of = of was not reported. risk = reported on potential of risk = size were performed and = size were not risk = size were Assessment of threats to construct validity It has been that preclinical to clinical may be to a mismatch between experimental conditions and the clinical disease the model is intended to represent (i.e. construct et al., 2013; and 2016). To evaluate clinical generalizability of the experimental conditions we performed a evaluation of construct validity using an that had been in a systematic review of preclinical sepsis (Table 4) et al., 2010). None of the experiments used animal models. Two experiments used animals with used of experiments used animal models did not report animal and used models of sepsis. of studies initiated MSC therapy after the induction of the disease to at the time of disease but none of the disease prior to initiating MSC therapy. studies used resuscitation while two of these studies also administered Two studies a of construct validity elements (i.e. at of there was no in effect size between these studies (OR 0.18, 95% CI and studies that fewer elements (OR 0.28, 95% CI (Figure 2—figure supplement Table 4 validity assessment of preclinical studies investigating the efficacy of mesenchymal stromal cells in models of sepsis. animal animal model of initiated after sepsis sepsis prior to initiating included included et al. (2009)A et al. et al. (2009) et al. (2010) et al. (2011) et al. (2012) et al. (2012) et al. (2012) et al. (2013) et al. (2013) et al. (2014) et al. (2014) et al. (2014) et al. (2014) et al. (2014) et al. (2014) et al. (2015) Legend: = = = author and year Mei that more than one was conducted in the = = = 6 mice = 6 mice = = = of = and puncture, live bacterial = administration = stromal cells administered after sepsis model = stromal cells administered at the time of sepsis to = stromal cells administered after of = stromal cells administered without a of = therapy from vehicle for cell = vehicle for cell administration or no of publication bias For the 20 demonstrated statistically significant effects of MSCs with a sample size of animals of a plot analysis of all experiments that publication bias exists (Figure which was by and analysis a relative of effect size of MSCs with a statistically significant reduction in mortality after (OR