Publications

While hardware generators have drastically improved design productivity, they have introduced new challenges for the task of verification. To effectively cover the functionality of a sophisticated generator, verification engineers require tools that provide the flexibility of metaprogramming. However, flexibility alone is not enough; components must also be portable in order to encourage the proliferation of verification libraries as well as enable new methodologies. This paper introduces fault, a Python embedded hardware verification language that aims to empower design teams to realize the full potential of generators.

Whilst some scientists study insects, molecules, brains, or clouds, other scientists study science itself. Meta-research, or “research-on-research”, is a burgeoning discipline that investigates efficiency, quality, and bias in the scientific ecosystem, topics that have become especially relevant amid widespread concerns about the credibility of the scientific literature. Meta-research may help calibrate the scientific ecosystem towards higher standards by providing empirical evidence that informs the iterative generation and refinement of reform initiatives. We introduce a translational framework that involves (1) identifying problems; (2) investigating problems; (3) developing solutions; and (4) evaluating solutions. In each of these areas, we review key meta-research endeavors and discuss several examples of prior and ongoing work. The scientific ecosystem is perpetually evolving; the discipline of meta-research presents an opportunity to use empirical evidence to guide its development and maximize its potential.

Confocal microscopy is a family of imaging techniques that employ focused patterned illumination and synchronized imaging to create cross-sectional views of 3D biological specimens. In this paper, we adapt confocal imaging to large-scale scenes by replacing the optical apertures used in microscopy with arrays of real or virtual video projectors and cameras. Our prototype implementation uses a video projector, a camera, and an array of mirrors. Using this implementation, we explore confocal imaging of partially occluded environments, such as foliage, and weakly scattering environments, such as murky water. We demonstrate the ability to selectively image any plane in a partially occluded environment, and to see further through murky water than is otherwise possible. By thresholding the confocal images, we extract mattes that can be used to selectively illuminate any plane in the scene.

Objective: To investigate whether the beneficial and harmful effects of platelet glycoprotein IIb/IIIa receptor blockers in non-ST elevation acute coronary syndromes (NSTE-ACS) depend on age. Methods: A meta-analysis of six trials of platelet glycoprotein IIb/IIIa receptor blockers in patients with NSTE-ACS (PRISM, PRISM-PLUS, PARAGON-A, PURSUIT, PARAGON-B, GUSTO IV-ACS; n = 31 402) was performed. We applied multivariable logistic regression analyses to evaluate the drug effects on death or non-fatal myocardial infarction at 30 days, and on major bleeding, by age subgroups (<60, 60–69, 70–79, ⩾80 years). We quantified the reduction of death or myocardial infarction as the number needed to treat (NNT), and the increase of major bleeding as the number needed to harm (NNH). Results: Subgroups had 11 155 (35%), 9727 (31%), 8468 (27%) and 2049 (7%) patients, respectively. The relative benefit of platelet glycoprotein IIb/IIIa receptor blockers did not differ significantly (p = 0.5) between age subgroups (OR (95% CI) for death or myocardial infarction: 0.86 (0.74 to 0.99), 0.90 (0.80 to 1.02), 0.97 (0.86 to 1.10), 0.90 (0.73 to 1.16); overall 0.91 (0.86 to 0.99). ORs for major bleeding were 1.9 (1.3 to 2.8), 1.9 (1.4 to 2.7), 1.6 (1.2 to 2.1) and 2.5 (1.5–4.1). Overall NNT was 105, and overall NNH was 90. The oldest patients had larger absolute increases in major bleeding, but also had the largest absolute reductions of death or myocardial infarction. Patients ⩾80 years had half of the NNT and a third of the NNH of patients <60 years. Conclusions: In patients with NSTE-ACS, the relative reduction of death or non-fatal myocardial infarction with platelet glycoprotein IIb/IIIa receptor blockers was independent of patient age. Larger absolute outcome reductions were seen in older patients, but with a higher risk of major bleeding. Close monitoring of these patients is warranted.

Genome-wide association studies (GWAS) have led to a rapid increase in available data on common genetic variants and phenotypes and numerous discoveries of new loci associated with susceptibility to common complex diseases. Integrating the evidence from GWAS and candidate gene studies depends on concerted efforts in data production, online publication, database development, and continuously updated data synthesis. Here the authors summarize current experience and challenges on these fronts, which were discussed at a 2008 multidisciplinary workshop sponsored by the Human Genome Epidemiology Network. Comprehensive field synopses that integrate many reported gene-disease associations have been systematically developed for several fields, including Alzheimer's disease, schizophrenia, bladder cancer, coronary heart disease, preterm birth, and DNA repair genes in various cancers. The authors summarize insights from these field synopses and discuss remaining unresolved issues—especially in the light of evidence from GWAS, for which they summarize empirical P-value and effect-size data on 223 discovered associations for binary outcomes (142 with P < 10−7). They also present a vision of collaboration that builds reliable cumulative evidence for genetic associations with common complex diseases and a transparent, distributed, authoritative knowledge base on genetic variation and human health. As a next step in the evolution of Human Genome Epidemiology reviews, the authors invite investigators to submit field synopses for possible publication in the American Journal of Epidemiology.

The UK National Institute for Health and Care Excellence (NICE) recently updated its recommendations for the treatment of depression1. This effort has many strengths, including the meticulous documentation of the process; systematic reviews, meta-analyses and cost-effectiveness analyses; and inclusion of stakeholder comments that feed into the guidelines. Here we attempt a constructive critical appraisal of areas where future improvements for this but also for other similar initiatives are feasible, with a special focus on psychotherapies for depression. We first notice that the methods and analyses of the NICE guidelines were not subjected to formal external peer review for any of the addressed questions. Asking stakeholders for comments is welcome, but it is unlikely to be equally rigorous, leaving it to the guideline committee how these comments are considered. External peer review is recommended as a default quality stan­dard for treatment guidelines2. Furthermore, study protocols were pre-registered only for some conditions (e.g., for new episodes of depression and treatment-resistant depression), but not for others (including chronic depression, depression with personality disorder, and psychotic depression). Pre-registering should be established as a default stan­dard in guidelines for all reviewed conditions. For the primary analysis concerning new episodes of depression, network meta-analysis (NMA) was chosen1. NMA has the advantage of incorporating both direct and indirect evidence, but complex assumptions need to be fulfilled, and the level of evidence provided is still debated3. For these reasons, NMA results and the derived inferences require extra caution. For treatment ranking, the guideline committee primarily focused on effect sizes from NMA treatment comparisons with placebo or treatment-as-usual, and compared these effect sizes between treatments. From these comparisons, the committee concluded that some treatments appeared to be "more effective" than others1. For most treatments, however, the differences between treatment and control effect sizes were below the minimal clinically significant difference defined by the committee (standardized mean difference, SMD >0.5 or <–0.5)1. This applies to comparisons between individual cognitive or cognitive-behavioral therapy (CT/CBT), individual interpersonal therapy (IPT), individual problem solving, individual short-term psychodynamic psychotherapy (STPP), and group behavior activation. Thus, with only subtle effect size differences, treatment ranking carries large uncertainty. Furthermore, assuming differences between two treatments if one of them shows descriptively a larger effect size than the other compared to a control condition, without comparing them directly, should be avoided4. The guideline committee reported head-to-head comparisons of active treatments only in a supplement. These comparisons show that, in more severe depression, the differences between individual behavioral therapy, individual CBT, individual IPT and individual STPP are neither statistically nor clinically significant (SMDs <0.50)1. In less severe depression, only a few clinically significant differences were found: for example, in a pairwise comparison, STPP was statistically and clinically significantly superior to counselling (SMD=–0.61, 95% CI: –1.05 to –0.17), but was ranked below counselling. Thus, the committee's conclusions about differences in efficacy between active treatments are not consistent with its own head-to-head comparisons. They are also not compatible with independent peer-reviewed evidence of no substantial differences in efficacy between psychotherapies5. The committee, however, erroneously interpreted this independent evidence5 as confirming its treatment ranking1,B, p.165. In summary, procedures for treatment ranking need to be pre-defined, and subtle differences below the threshold of clinically meaningful values should not be overstated. In principle, possible allegiance and conflicts of interests need to be controlled for2, for example by including methodologists, patients, and different-field experts, and by limiting the involvement of field specialists to a consultation role6. Avoidance of stacking is also essential, ensuring that guideline developers do not have an over-representation of believers in one or another treatment modality6. The guideline committee based the hierarchy of treatment recommendations on both efficacy and cost-effectiveness, which is useful in trying to optimize the use of treatments for conditions with high prevalence1. For cost-effectiveness, however, peer reviews and pre-registration are missing. Moreover, the cost-effectiveness literature is notoriously replete with biases. This further complicates matters in a field such as depression where the primary studies are often also biased (e.g., sponsor bias in pharmacotherapy trials and allegiance bias in psychotherapy trials). Furthermore, the studies used by the committee for cost-effectiveness analysis did not cover all relevant treatment types. For those not covered, it is not clear whether cost-effectiveness estimates are valid. Additional cost-effectiveness analyses commissioned by the committee were based on the NMA treatment-control effect sizes shown above to be questionable, which further limits the derived treatment ranking. Another challenge is whether extrapolations from new episodes of depression to other conditions are valid, when there is no solid evidence for these other categories of depression. For example, in depression with personality disorder, the committee recommends combining antidepressants and psychotherapy. For the choice between psychotherapies, readers are referred to the treatments for new episodes of depression. Then, for patients not sufficiently responding to pharmacotherapy alone, switching to psychotherapies listed for new episodes of more severe depression is recommended as one option. In reviewing new episodes of depression, however, the committee excluded depression with personality disorder and treatment-resistant depression. Thus, the committee's ranking of psychotherapies for new episodes of depression may not be valid for these other conditions. Finally, for the cost-effectiveness of chronic depression and depression with personality disorder, the committee also used the economic data for new episodes of depression. As another problem, the guideline committee found the quality of studies to be quite low. The committee tried to adjust results for bias, but a pre-registered threshold analysis for assessing confidence in recommendations was not carried out. Quality of evidence was evaluated narratively using the GRADE system, but without assessing confidence. Assessing confidence in evidence is essential for guidelines6. The committee also draws an arbitrary distinction between the more complex forms of depression, which not only reduces gener­alizability to clinical practice but appears to have led to the ex­clusion of relevant studies. Available randomized controlled tri­als have not clearly distinguished between chronic depression and treat­ment-resistant depression. For chronic depression, the committee recommends CBT, antidepressants or their combination1. However, these recommendations do not take into account the evidence for STPP and long-term psychodynamic therapy in treat­ment-resistant depression and in depression with personality disorder7, 8, conditions highly associated with chronic depression. Guidelines need to avoid arbitrary distinctions of disorders. Moreover, the committee did not sufficiently consider the limitations of the available evidence2, especially the limited remission rates (about 30%) of short-term psychotherapies (4-20 sessions), with SMDs of 0.309. Aggravating this problem, most effect sizes of short-term treatments are not stable at follow-up1. Especially for chronic depression, success rates may be improved with longer-term treatments9. The committee, however, considered long-term treatments only as an option for depression with personality disorder. Finally, an explicit link between evidence and recommendations is missing2. We acknowledge that the evidence in this field is uncertain, and this may be the reason why the committee found it "difficult… to link the recommendations directly to the NMA results"1,B, pp.48,66, and based its recommendations ultimately on "clinical experience"1,B, p.66. However, it is unclear whether clinical experience can offer any solid guidance when treatment differences are modest, uncertainty is high and bias is substantial. Guidelines should fully admit this uncertainty and avoid over-simplified, over-confident recommendations6.