Home Publications Research Grants Inventions & Patents Awards Additional Research Activities Faculties & Institutions Research Areas

Is crowd-algorithm collaboration an advanced alternative to crowd-sourcing on cytology slides?

Marzahl C, Aubreville M, Bertram CA, Gerlach S, Maier J, Voigt J, Hill J, Klopfleisch R, Maier A (2020)

---

Publication Type: Conference contribution

Publication year: 2020

Journal

Publisher: Springer

Pages Range: 26-31

Conference Proceedings Title: Informatik aktuell

Event location: Berlin

ISBN: 9783658292669

DOI: 10.1007/978-3-658-29267-6_5

Abstract

Modern, state-of-the-art deep learning approaches yield human like performance in numerous object detection and classification tasks. The foundation for their success is the availability of training datasets of substantially high quantity, which are expensive to create, especially in the field of medical imaging. Crowdsourcing has been applied to create large datasets for a broad range of disciplines. This study aims to explore the challenges and opportunities of crowd-algorithm collaboration for the object detection task of grading cytology whole slide images. We compared the classical crowdsourcing performance of twenty participants with their results from crowd-algorithm collaboration. All participants performed both modes in random order on the same twenty images. Additionally, we introduced artificial systematic flaws into the precomputed annotations to estimate a bias towards accepting precomputed annotations. We gathered 9524 annotations on 800 images from twenty participants organised into four groups in concordance to their level of expertise with cytology. The crowd-algorithm mode improved on average the participants’ classification accuracy by 7%, the mean average precision by 8% and the inter-observer Fleiss’ kappa score by 20%, and reduced the time spent by 31%. However, two thirds of the artificially modified false labels were not recognised as such by the contributors. This study shows that crowd-algorithm collaboration is a promising new approach to generate large datasets when it is ensured that a carefully designed setup eliminates potential biases.

Authors with CRIS profile

Involved external institutions

How to cite

APA:

Marzahl, C., Aubreville, M., Bertram, C.A., Gerlach, S., Maier, J., Voigt, J.,... Maier, A. (2020). Is crowd-algorithm collaboration an advanced alternative to crowd-sourcing on cytology slides? In Thomas Tolxdorff, Thomas M. Deserno, Heinz Handels, Andreas Maier, Klaus H. Maier-Hein, Christoph Palm (Eds.), Informatik aktuell (pp. 26-31). Berlin, DE: Springer.

MLA:

Marzahl, Christian, et al. "Is crowd-algorithm collaboration an advanced alternative to crowd-sourcing on cytology slides?" Proceedings of the International workshop on Algorithmen - Systeme - Anwendungen, 2020, Berlin Ed. Thomas Tolxdorff, Thomas M. Deserno, Heinz Handels, Andreas Maier, Klaus H. Maier-Hein, Christoph Palm, Springer, 2020. 26-31.

BibTeX: Download