State space techniques have proven to be useful for measuring and improving the coverage of test vectors that are used during functional validation via simulation. By comparing the state and edge coverage provided by tests with that which is possible in the design's state graph, the designer can estimate how well tested the design is and identify areas that need better testing. Unfortunately, for many interesting designs, the full state graph may be too large to fully explore, or if it is explorable, the resulting coverage may be so low as to provide limited feedback. Several techniques have been proposed that identify and work with an interesting subset of the design's state machines, but they still require computing the full state graph before projecting it. In this paper we discuss projection directed state exploration, in which a projection from the full graph is found while exploring only the relevant portion of the full graph. Even with this limited exploration, BDD size blowup is still a problem. To deal with this, we have also developed several interactive tools that provide feedback to the designer, and allow them to add hints to help with the exploration.
Meta-analyses play an important role in synthesizing evidence from diverse studies and datasets that address similar questions. A major obstacle for meta-analyses arises from biases in reporting. In particular, it is speculated that findings which do not achieve formal statistical significance are less likely reported than statistically significant findings. Moreover, the patterns of bias can be complex and may also depend on the timing of the research results and their relationship with previously published work. In this paper, we present an approach that is specifically designed to analyze large-scale datasets on published results. Such datasets are currently emerging in diverse research fields, particularly in molecular medicine. We use our approach to investigate a dataset on Alzheimer's disease (AD) that covers 1167 results from case-control studies on 102 genetic markers. We observe that initial studies on a genetic marker tend to be substantially more biased than subsequent replications. The chances for initial, statistically non-significant results to be published are estimated to be about 44% (95% CI, 32% to 63%) relative to statistically significant results, while statistically non-significant replications have almost the same chance to be published as statistically significant replications (84%; 95% CI, 66% to 107%). Early replications tend to be biased against initial findings, an observation previously termed Proteus phenomenon: The chances for non-significant studies going in the same direction as the initial result are estimated to be lower than the chances for non-significant studies opposing the initial result (73%; 95% CI, 55% to 96%). Such dynamic patterns in bias are difficult to capture by conventional methods, where typically simple publication bias is assumed to operate. Our approach captures and corrects for complex dynamic patterns of bias, and thereby helps generating conclusions from published results that are more robust against the presence of different coexisting types of selective reporting.
This study assesses data sharing status and lifting of embargoes in randomized clinical trials from top medical journals 3 to 5 years after publication.
An estimated 1.1 billion people currently smoke cigarettes,<sup>1</sup> and 50 to 70% likely will die from tobacco-related causes.<sup>2</sup> This translates to 550 to 770 million expected tobacco deaths among those who currently smoke. Many additional deaths will accrue in successive generations if the status quo continues. Of interest is the reversibility of the excess mortality risk of smoking. The meta-analysis by Cho et al.<sup>3</sup> of four large national cohorts of nearly 1.5 million adults followed on average 14.8 years yielded 23.0 million person-years of observational data with over 120,000 deaths identified through linked death registries.
While some independent replication of meta-analyses by different teams is possibly useful, the overall picture suggests that there is a waste of efforts with many topics covered by multiple overlapping meta-analyses.
Share your important work on peer review, publication, and the conduct of scientific research
Three insightful papers in this issue (1-3) highlight different types of biases and errors in obesity research and related fields of investigation. George et al. (1) review a wide variety of errors and biases in misuse of statistical methods, misconceptions in scientific inference, improper or inadequate consideration of multiplicity, and suboptimal or selective reporting. Johns et al. (2) provide a meta-epidemiological assessment of data from control groups from 29 randomized trials of obesity and show that participants in inactive control groups spuriously seem to lose weight after 12 months—an extra reason why non-controlled studies should be less trusted. Fontaine et al. (3) discuss the subtleties of placebo effects and how placebo-related factors may cause an effect even among people who know that they are receiving a placebo. Errors and biases are by no means unique to obesity research. They pervade all fields of scientific investigation (4). However, obesity and related fields, in particular nutrition research, have received an extra share of attention on these issues. There are probably many reasons for this, including the unquestionable importance of the subject matter; the extra interest for this topic (especially the "bad news") by mass media (5); the refutations of scores of epidemiological associations especially in nutrition research (6); the large volume of papers published; the persistence of some myths based on no evidence (7, 8) which distort further investigation by offering a wrong starting point; the routine use of measurement tools with high error rates and serious biases (e.g., nutrition questionnaires based on self-reporting) (9); the relative lack of transparency (e.g., no pre-specified protocols, registration, data sharing); and the recalcitrance of some segments of this literature to adopt standard practices (e.g., proper adjustment for multiple comparisons and more stringent statistical thresholds) that might have saved several embarrassments. Some fields in observational research in particular have seen the massive propagation of the practice of "salami slicing," where dozens of investigators co-author papers practically reporting a single association at a time and where each paper consists of a tiny part of a much larger data dredging agenda from the same data set "gold mine." In observational research, some data sets have already published hundreds of papers (10) instead of a couple dozen that would have been more appropriate, e.g., if they had used exposure-wide association approaches that can evaluate dozens and hundreds of exposures within the same analysis (11, 12). Salami slicing is also seen increasingly in some randomized trials of nutrition and obesity, where multiple publications of outcomes and analyses stem from the same trial. For example, a search in PubMed (December 17, 2015) with PREDIMED [ti] or "PREDIMED Study Investigators" retrieves 95 papers. Even though this is an excellent, pivotal randomized trial, one can question how much data dredging a single trial can tolerate. For example, a recent PREDIMED paper (13) shows a beneficial effect for invasive breast cancer incidence in the group of women with high cardiovascular risk. This outcome is claimed to have been pre-specified, but this is very hard to believe, unless hundreds of outcomes and analyses were pre-specified. At the same time, a lot of high-quality research is also conducted in obesity-related themes, and the research questions remain highly relevant and of major public health consequence. One has to decide what to do with all these potential errors and biases and how one can improve further the credibility of the wider research agenda on obesity. At a first step, errors and biases should be identified. Some problems (e.g., data dredging and hidden multiplicity) are well documented both about their severity and high prevalence, while others (e.g., placebo-related factors) are far less studied and/or may pertain only to special circumstances and study designs. Getting a better sense of the prevalence of each type of error and bias can be informative. This may sensitize investigators and statisticians who design, conduct, and report a study; reviewers and editors; and funders who contemplate funding new studies. Beyond identifying problems, it is important to correct them. For most of these biases and errors, corrective action is more efficient early, in the design phase. Major, common problems that can be tackled to a good extent include consideration of multiplicity, proper controls, and use of randomization whenever possible. The continued reliance on uncontrolled studies to test obesity interventions is precarious. The agenda of randomized controlled trials should be promoted and enhanced with larger trials, improved fidelity of the interventions, and pre-registration of outcomes and main analyses (14). Improved reporting and more appropriate statistical inferences can also help improve this research agenda. Conversely, some biases are more difficult to correct even with good intentions. For example, confirmation bias is particularly difficult to eradicate. Therefore, one may need to interpret results cautiously for these types of more recalcitrant biases. There are a few situations where biases may even need to be explicitly endorsed according to a carefully thought-out plan. For example, as Fontaine et al. highlight (3), placebo-related factors such as information disclosure, expectations, conditioning, and empathy should be avoided when one is designing controlled studies that try to evaluate the effectiveness of diet or other lifestyle interventions, but they may need to be endorsed in studies that assess real-life effectiveness upon implementing interventions in real-world, everyday clinical encounters. Finally, some segments of the obesity and nutrition research agenda may simply have to be abandoned, to free more resources for understudied research pathways and for strengthening the design of proper, more bias-proof studies. For example, the continuous production of thousands of papers of observational epidemiology that assess one nutrient at a time in association with one outcome has reached the point of even being ridiculed by hoaxes, as in the recent chocolate and weight loss hoax (http://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800). When it is known, after thousands of published papers, that effect sizes are expected to be tiny, observational studies will be unable to eliminate noise to a point that offers reasonable certainty about the validity of observed results. Continuing to use a nail and a hammer in the same way is not a wise investment of resources, especially when there are many other serious scientific questions to tackle in this important discipline.
A compact low-cost laser interferometer system for measuring linear displacements with sub-micrometer resolution is described and first performance evaluations reported. The four-photo-diode detectors system can also provide a sensitive, simultaneous indication of straightness in two axes
The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large. History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that “it is obvious that progress is made” is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.
