First Steps to Evaluate an NLP Tool's Medication Extraction Accuracy from Discharge Letters

Caliskan D, Zierk J, Kraska D, Schulz S, Daumke P, Prokosch HU, Kapsner L (2021)


Publication Type: Book chapter / Article in edited volumes

Publication year: 2021

Publisher: IOS Press

Edited Volumes: German Medical Data Sciences: Bringing Data to Life

Series: Studies in Health Technology and Informatics

Book Volume: 278

Pages Range: 224-230

ISBN: 978-1-64368-177-1

DOI: 10.3233/SHTI210073

Abstract

INTRODUCTION: The aim of this study is to evaluate the use of a natural language processing (NLP) software to extract medication statements from unstructured medical discharge letters. METHODS: Ten randomly selected discharge letters were extracted from the data warehouse of the University Hospital Erlangen (UHE) and manually annotated to create a gold standard. The AHD NLP tool, provided by MIRACUM's industry partner was used to annotate these discharge letters. Annotations by the NLP tool where then compared to the gold standard on two levels: phrase precision (whether or not the whole medication statement has been identified correctly) and token precision (whether or not the medication name has been identified correctly within correctly discovered medication phrases). RESULTS: The NLP tool detected medication related phrases with an overall F-measure of 0.852. The medication name has been identified correctly with an overall F-measure of 0.936. DISCUSSION: This proof-of-concept study is a first step towards an automated scalable evaluation system for MIRACUM's industry partner's NLP tool by using a gold standard. Medication phrases and names have been correctly identified in most cases by the NLP system. Future effort needs to be put into extending and validating the gold standard.

Authors with CRIS profile

Involved external institutions

How to cite

APA:

Caliskan, D., Zierk, J., Kraska, D., Schulz, S., Daumke, P., Prokosch, H.-U., & Kapsner, L. (2021). First Steps to Evaluate an NLP Tool's Medication Extraction Accuracy from Discharge Letters. In Rainer Röhrig, Tim Beißbarth, Werner Brannath, Hans-Ulrich Prokosch, Irene Schmidtmann, Susanne Stolpe, Antonia Zapf (Eds.), German Medical Data Sciences: Bringing Data to Life. (pp. 224-230). IOS Press.

MLA:

Caliskan, Deniz, et al. "First Steps to Evaluate an NLP Tool's Medication Extraction Accuracy from Discharge Letters." German Medical Data Sciences: Bringing Data to Life. Ed. Rainer Röhrig, Tim Beißbarth, Werner Brannath, Hans-Ulrich Prokosch, Irene Schmidtmann, Susanne Stolpe, Antonia Zapf, IOS Press, 2021. 224-230.

BibTeX: Download