Medical Information Extraction with NLP-Powered QABots: a Real-World Scenario

The advent of computerized medical recording systems in healthcare facilities has made data retrieval tasks easier, compared with manual recording. Nevertheless, the potential of the information contained within medical records remains largely untapped, mostly due to the time and effort required to extract data from unstructured documents. Natural Language Processing (NLP) represents a promising solution to this challenge, as it enables the use of automated text-mining tools for clinical practitioners. In this work, we present the architecture of the Virtual Dementia Institute (IVD), a consortium of sixteen Italian hospitals, using the NLP Extraction and Management Tool (NEMT), a (semi-)automated end-to-end pipeline that extracts relevant information from clinical documents and stores it in a centralized database. NEMT core is a Question Answering Bot (QABot) based on a modern NLP model, fine-tuned using thousands of examples produced from IVD centers. Detailed descriptions of the process for defining a common minimum dataset, the Inter-Annotator Agreement calculated on clinical documents, and NEMT results, are provided. The best QABot performance in terms of Exact Match (EM) and F1-score (78.1% and 84.7%) outperforms ChatGPTv3.5 (68.9% and 52.5%). NEMT represents an efficient tool that paves the way for medical information extraction and exploitation for new research studies.