MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images

Zimmerer D, Full PM, Isensee F, Jaeger P, Adler T, Petersen J, Koehler G, Ross T, Reinke A, Kascenas A, Jensen BS, O'Neil AQ, Tan J, Hou B, Batten J, Qiu H, Kainz B, Shvetsova N, Fedulova I, Dylov D, Yu B, Zhai J, Hu J, Si R, Zhou S, Wang S, Li X, Chen X, Zhao Y, Marimont SN, Tarroni G, Saase V, Maier-Hein L, Maier-Hein K (2022)

Publication Type: Journal article

Publication year: 2022

Journal

IEEE Transactions on Medical Imaging Institute of Electrical and Electronics Engineers (IEEE)

Book Volume: 41

Pages Range: 2728-2738

Journal Issue: 10

DOI: 10.1109/TMI.2022.3170077

Abstract

Detecting Out-of-Distribution (OoD) data is one of the greatest challenges in safe and robust deployment of machine learning algorithms in medicine. When the algorithms encounter cases that deviate from the distribution of the training data, they often produce incorrect and over-confident predictions. OoD detection algorithms aim to catch erroneous predictions in advance by analysing the data distribution and detecting potential instances of failure. Moreover, flagging OoD cases may support human readers in identifying incidental findings. Due to the increased interest in OoD algorithms, benchmarks for different domains have recently been established. In the medical imaging domain, for which reliable predictions are often essential, an open benchmark has been missing. We introduce the Medical-Out-Of-Distribution-Analysis-Challenge (MOOD) as an open, fair, and unbiased benchmark for OoD methods in the medical imaging domain. The analysis of the submitted algorithms shows that performance has a strong positive correlation with the perceived difficulty, and that all algorithms show a high variance for different anomalies, making it yet hard to recommend them for clinical practice. We also see a strong correlation between challenge ranking and performance on a simple toy test set, indicating that this might be a valuable addition as a proxy dataset during anomaly detection algorithm development.

Authors with CRIS profile

Bernhard Kainz Professur für Image Data Exploration and Analysis

Involved external institutions

Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB) Deutsches Krebsforschungszentrum (DKFZ) DE Germany (DE) National University of Defense Technology CN China (CN) City, University of London GB United Kingdom (GB) University of Edinburgh GB United Kingdom (GB) Skolkovo Institute of Science and Technology RU Russian Federation (RU) University of Glasgow GB United Kingdom (GB) Ruprecht-Karls-Universität Heidelberg DE Germany (DE)

How to cite

APA:

Zimmerer, D., Full, P.M., Isensee, F., Jaeger, P., Adler, T., Petersen, J.,... Maier-Hein, K. (2022). MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images. IEEE Transactions on Medical Imaging, 41(10), 2728-2738. https://dx.doi.org/10.1109/TMI.2022.3170077

MLA:

Zimmerer, David, et al. "MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images." IEEE Transactions on Medical Imaging 41.10 (2022): 2728-2738.

BibTeX: Download