Can Non-specialists Provide High Quality Gold Standard Labels in Challenging Modalities?

Budd S, Day TG, Simpson JM, Lloyd K, Matthew J, Skelton E, Razavi R, Kainz B (2021)

Publication Status: Published

Publication Type: Authored book, Volume of book series

Publication year: 2021

Publisher: Springer Science and Business Media Deutschland GmbH

Pages Range: 251-262

ISBN: 9783030877217

DOI: 10.1007/978-3-030-87722-4_23

Open Access Link: https://arxiv.org/abs/2107.14682

Abstract

Probably yes.—Supervised Deep Learning dominates performance scores for many computer vision tasks and defines the state-of-the-art. However, medical image analysis lags behind natural image applications. One of the many reasons is the lack of well annotated medical image data available to researchers. One of the first things researchers are told is that we require significant expertise to reliably and accurately interpret and label such data. We see significant inter- and intra-observer variability between expert annotations of medical images. Still, it is a widely held assumption that novice annotators are unable to provide useful annotations for use by clinical Deep Learning models. In this work we challenge this assumption and examine the implications of using a minimally trained novice labelling workforce to acquire annotations for a complex medical image dataset. We study the time and cost implications of using novice annotators, the raw performance of novice annotators compared to gold-standard expert annotators, and the downstream effects on a trained Deep Learning segmentation model’s performance for detecting a specific congenital heart disease (hypoplastic left heart syndrome) in fetal ultrasound imaging.

Authors with CRIS profile

Bernhard Kainz Professur für Image Data Exploration and Analysis

Involved external institutions

King’s College London GB United Kingdom (GB) Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB)

How to cite

APA:

Budd, S., Day, T.G., Simpson, J.M., Lloyd, K., Matthew, J., Skelton, E.,... Kainz, B. (2021). Can Non-specialists Provide High Quality Gold Standard Labels in Challenging Modalities? Springer Science and Business Media Deutschland GmbH.

MLA:

Budd, Samuel, et al. Can Non-specialists Provide High Quality Gold Standard Labels in Challenging Modalities? Springer Science and Business Media Deutschland GmbH, 2021.

BibTeX: Download