Benchmarking Human–AI collaboration for common evidence appraisal tools

Tim Woelfle; Julian Hirt; Perrine Janiaud; Ludwig Kappos; John P A Ioannidis; Lars G. Hemkens

doi:10.1016/j.jclinepi.2024.111533

Abstract

1 min read

Current LLMs alone appraised evidence worse than humans. Human-AI collaboration may reduce workload for the second human rater for the assessment of reporting (PRISMA) and methodological rigor (AMSTAR) but not for complex tasks such as PRECIS-2.

Related publications

Preprint2024

Benchmarking Human-AI Collaboration for Common Evidence Appraisal Tools

Tim Woelfle, Julian Hirt, Perrine Janiaud, Ludwig Kappos, John P A Ioannidis, Lars G. Hemkens

Article2026

Embodied Digital Therapists with LLM Personalization for Aphasia Rehabilitation: Characterizing Human-AI Collaboration Boundaries

Mei Yu, Lifeng Zhu, Wenli Chen, Jin Liu, Zhaoyi Liu, Aiguo Song

Article2024

A framework for human–robot collaboration enhanced by preference learning and ergonomics

Matteo Meregalli Falerni, Vincenzo Pomponi, Hamid Reza Karimi, Matteo Lavit Nicora, Le Anh Dao, Matteo Malosio, Loris Roveda

Robotics and Computer-Integrated Manufacturing

Preprint2025

Treatment Approaches for Problematic Usage of the Internet (PUI): A Dual-Level Meta-Analysis of Meta-Analyses and Randomized Controlled Trials

Alireza Valyan, Fateme Sadat Abolghasemi, Sophia Achab, Mitra Ashrafi, Alexander Baldacchino, Zsolt Demetrovics, Naomi Fineberg, Parastoo Ghorbani, Yasser Khazaal, Kristiana Siste, Anise M. S. Wu, Mehran Zare-Bidoky, Dan Joseph Stein, Marc N. Potenza, Hamed Ekhtiari, Hamed Ekhtiari

Article2023

A novel system-theoretic approach for human-system collaboration safety: Case studies on two degrees of autonomy for autonomous ships

Tingting Cheng, Ingrid B. Utne, Bing Wu, Qing Wu

Reliability Engineering & System Safety