Odin’s Eye – A Close Look at Gamification of Labelling of Ophthalmic Diseases

Fan F, Qiu J, Wang J, Valianos S, Farooq MM, Fu W, Kordon FJ, Maier A (2020)


Publication Language: English

Publication Type: Conference contribution, Abstract of lecture

Publication year: 2020

Event location: Online Streaming

Abstract

Introduction

State of the art deep learning [1] requires vast amounts of accurately labeled training data to enable high classification performance [2]. In order to obtain sufficient amounts of data, data donation is a feasible approach [3]. Yet, the data is only usable, if correct annotations are present. One way to create such annotations is crowd-sourcing via gamification. In this paper, we present an integrated training and annotation approach that allows large scale annotation of ophthalmic diseases.


Methods

We created a game called “Odin’s Eye” in order to make image classification an exciting and rewarding experience. The game has three modes that are used to slowly lead the player to the complex field of ophthalmic diseases. In order to do so, we used image data from the ODIR 2019 Challenge (https://odir2019.grand-challenge.org/dataset/). The dataset contains more than 7000 images showing healthy eye data and different pathologies. 

In “Normal Mode”, the player is shown one fundus image and is asked to select whether the shown image is “normal” or shows Cataract, Macular Degeneration, Glaucoma, Retinopathy, or Myopia. After a sufficient number of correct answers, the player can access the unlabeled mode, in which the user is able to give annotation to unlabeled image. The annotations for the unlabeled images will be shown in the “Unlabeled Result”.

The “Difficult Mode” is inspired by puzzle games such as Candy Crush or Zookeeper. Here, the player is displayed various fundus images and is asked to align them such that three images showing the same pathology form either a row or a column. Once such a triplet is found, it is eliminated and new images enter the game canvas from the top.  In order to make the game more challenging, a curtain enters the field of view from the top that gradually increases the pressure on the player as well as increases game difficulty via decreasing the image brightness. After each successful elimination, the curtain is raised a little and the time limit is increased. Rewarding strategies like reward animation and image shuffling are applied to achieve better user experience. The idea of using triplets is appealing as images of unknown pathology can be mixed in the game area. The player will implicitly classify such images when he tries to align them with two more images of the same class.

A leader board shows high scores in order to encourage players to compare their scores online and to compete in creating a large number of annotations.


Results

A prototype was implemented in Unity [4]. In order to test the game idea, only five prototypical images were chosen for each pathology at present. The difficulty of the game can easily be increased by increasing types of diseases. First experiences with test players confirmed that in particular the “Difficult Mode” has a very rewarding game experience. Prototypes are available as Android APK and WebGL (https://www.medicaldatadonors.org/index.php/odins-eye/). An in-game video was created to demonstrate the gameplay (https://youtu.be/UehnND9gkvY).

 

Conclusion

A believe that we created a challenging yet rewarding game experience. In future versions, we will make use of additional images from the ODIR 2019 dataset in order to create slowly increasing difficulty in the game to convince even expert players to keep playing Odin’s Eye.


References

[1] Maier, A., Syben, C., Lasser, T., & Riess, C. (2019). A gentle introduction to deep learning in medical image processing. Zeitschrift für Medizinische Physik, 29(2), 86-101.

[2] Bertram, C. A., Aubreville, M., Marzahl, C., Maier, A., & Klopfleisch, R. (2019). A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Scientific data, 6(1), 1-9.

[3] Servadei, L., Schmidt, R., Eidelloth, C., & Maier, A. (2017, October). Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems" (pp. 87-97). Springer, Cham.

[4] Murray, J. W. (2014). C# game programming cookbook for Unity 3D. AK Peters/CRC Press. 

Authors with CRIS profile

Additional Organisation(s)

How to cite

APA:

Fan, F., Qiu, J., Wang, J., Valianos, S., Farooq, M.M., Fu, W.,... Maier, A. (2020). Odin’s Eye – A Close Look at Gamification of Labelling of Ophthalmic Diseases. Paper presentation at Joint Conference of the GMDS & CEN-IBS 2020, Online Streaming.

MLA:

Fan, Fuxin, et al. "Odin’s Eye – A Close Look at Gamification of Labelling of Ophthalmic Diseases." Presented at Joint Conference of the GMDS & CEN-IBS 2020, Online Streaming 2020.

BibTeX: Download