Biases in obesity research: Identify, correct, endorse, or abandon effort?

Three insightful papers in this issue (1-3) highlight different types of biases and errors in obesity research and related fields of investigation. George et al. (1) review a wide variety of errors and biases in misuse of statistical methods, misconceptions in scientific inference, improper or inadequate consideration of multiplicity, and suboptimal or selective reporting. Johns et al. (2) provide a meta-epidemiological assessment of data from control groups from 29 randomized trials of obesity and show that participants in inactive control groups spuriously seem to lose weight after 12 months—an extra reason why non-controlled studies should be less trusted. Fontaine et al. (3) discuss the subtleties of placebo effects and how placebo-related factors may cause an effect even among people who know that they are receiving a placebo. Errors and biases are by no means unique to obesity research. They pervade all fields of scientific investigation (4). However, obesity and related fields, in particular nutrition research, have received an extra share of attention on these issues. There are probably many reasons for this, including the unquestionable importance of the subject matter; the extra interest for this topic (especially the "bad news") by mass media (5); the refutations of scores of epidemiological associations especially in nutrition research (6); the large volume of papers published; the persistence of some myths based on no evidence (7, 8) which distort further investigation by offering a wrong starting point; the routine use of measurement tools with high error rates and serious biases (e.g., nutrition questionnaires based on self-reporting) (9); the relative lack of transparency (e.g., no pre-specified protocols, registration, data sharing); and the recalcitrance of some segments of this literature to adopt standard practices (e.g., proper adjustment for multiple comparisons and more stringent statistical thresholds) that might have saved several embarrassments. Some fields in observational research in particular have seen the massive propagation of the practice of "salami slicing," where dozens of investigators co-author papers practically reporting a single association at a time and where each paper consists of a tiny part of a much larger data dredging agenda from the same data set "gold mine." In observational research, some data sets have already published hundreds of papers (10) instead of a couple dozen that would have been more appropriate, e.g., if they had used exposure-wide association approaches that can evaluate dozens and hundreds of exposures within the same analysis (11, 12). Salami slicing is also seen increasingly in some randomized trials of nutrition and obesity, where multiple publications of outcomes and analyses stem from the same trial. For example, a search in PubMed (December 17, 2015) with PREDIMED [ti] or "PREDIMED Study Investigators" retrieves 95 papers. Even though this is an excellent, pivotal randomized trial, one can question how much data dredging a single trial can tolerate. For example, a recent PREDIMED paper (13) shows a beneficial effect for invasive breast cancer incidence in the group of women with high cardiovascular risk. This outcome is claimed to have been pre-specified, but this is very hard to believe, unless hundreds of outcomes and analyses were pre-specified. At the same time, a lot of high-quality research is also conducted in obesity-related themes, and the research questions remain highly relevant and of major public health consequence. One has to decide what to do with all these potential errors and biases and how one can improve further the credibility of the wider research agenda on obesity. At a first step, errors and biases should be identified. Some problems (e.g., data dredging and hidden multiplicity) are well documented both about their severity and high prevalence, while others (e.g., placebo-related factors) are far less studied and/or may pertain only to special circumstances and study designs. Getting a better sense of the prevalence of each type of error and bias can be informative. This may sensitize investigators and statisticians who design, conduct, and report a study; reviewers and editors; and funders who contemplate funding new studies. Beyond identifying problems, it is important to correct them. For most of these biases and errors, corrective action is more efficient early, in the design phase. Major, common problems that can be tackled to a good extent include consideration of multiplicity, proper controls, and use of randomization whenever possible. The continued reliance on uncontrolled studies to test obesity interventions is precarious. The agenda of randomized controlled trials should be promoted and enhanced with larger trials, improved fidelity of the interventions, and pre-registration of outcomes and main analyses (14). Improved reporting and more appropriate statistical inferences can also help improve this research agenda. Conversely, some biases are more difficult to correct even with good intentions. For example, confirmation bias is particularly difficult to eradicate. Therefore, one may need to interpret results cautiously for these types of more recalcitrant biases. There are a few situations where biases may even need to be explicitly endorsed according to a carefully thought-out plan. For example, as Fontaine et al. highlight (3), placebo-related factors such as information disclosure, expectations, conditioning, and empathy should be avoided when one is designing controlled studies that try to evaluate the effectiveness of diet or other lifestyle interventions, but they may need to be endorsed in studies that assess real-life effectiveness upon implementing interventions in real-world, everyday clinical encounters. Finally, some segments of the obesity and nutrition research agenda may simply have to be abandoned, to free more resources for understudied research pathways and for strengthening the design of proper, more bias-proof studies. For example, the continuous production of thousands of papers of observational epidemiology that assess one nutrient at a time in association with one outcome has reached the point of even being ridiculed by hoaxes, as in the recent chocolate and weight loss hoax (http://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800). When it is known, after thousands of published papers, that effect sizes are expected to be tiny, observational studies will be unable to eliminate noise to a point that offers reasonable certainty about the validity of observed results. Continuing to use a nail and a hammer in the same way is not a wise investment of resources, especially when there are many other serious scientific questions to tackle in this important discipline.