A Divide-and-Conquer Approach towards Understanding Deep Networks - Fu, Weilin; Breininger, Katharina; Schaffert, Roman; Ravikumar, Nishant; Maier, Andreas (Medical Image Computing and Computer Assisted Intervention (MICCAI 2019), 2019)
Deep neural networks have achieved tremendous success in various fields including medical image segmentation. However, they have long been criticized for being a black-box, in that interpretation, understanding and correcting architectures is difficult as there is no general theory for deep neural network design. Previously, precision learning was proposed to fuse deep architectures and traditional approaches. Deep networks constructed in this way benefit from the original known operator, have fewer parameters, and improved interpretability. However, they do not yield state-of-the-art performance in all applications. In this paper, we propose to analyze deep networks using known operators, by adopting a divide-and-conquer strategy to replace network components, whilst retaining networks performance. The task of retinal vessel segmentation is investigated for this purpose. We start with a high-performance U-Net and show by step-by-step conversion that we are able to divide the network into modules of known operators. The results indicate that a combination of a trainable guided filter and a trainable version of the Frangi filter yields a performance at the level of U-Net (AUC 0.974 vs. 0.972) with a tremendous reduction in parameters (111, 536 vs. 9, 575). In addition, the trained layers can be mapped back into their original algorithmic interpretation and analyzed using standard tools of signal processing. <
Keywords: Precision Learning; Debugging; CNN

Analyzing an Imitation Learning Network for Fundus Image Registration Using a Divide-and-Conquer Approach - Bayer, Siming; Zhong, Xia; Fu, Weilin; Ravikumar, Nishant; Maier, Andreas (Algorithmen – Systeme – Anwendungen. Proceedings des Workshops, 2020)
Comparison of microvascular circulation on fundoscopic images is a non-invasive clinical indication for the diagnosis and monitoring of diseases, such as diabetes and hypertensions. The differences between intra-patient images can be assessed quantitatively by registering serial acquisitions. Due to the variability of the images (i.e. contrast, luminosity) and the anatomical changes of the retina, the registration of fundus images remains a challenging task. Recently, several deep learning approaches have been proposed to register fundus images in an end-to-end fashion, achieving remarkable results.However, the results are difficult to interpret and analyze. In this work, we propose an imitation learning framework for the registration of 2D color funduscopic images for a wide range of applications such as disease monitoring, image stitching and super-resolution. We follow a divide-and-conquer approach to improve the interpretability of the proposed network, and analyze both the influence of the input image and the hyperparameters on the registration result.The results show that the proposed registration network reduces the initial target registration error up to 95 %<br

Degenerating u-net on retinal vessel segmentation: What do we really need? - Fu, Weilin; Breininger, Katharina; Pan, Zhaoya; Maier, Andreas (Informatik aktuell, 2020)
Retinal vessel segmentation is an essential step for fundus image analysis. With the recent advances of deep learning technologies, many convolutional neural networks have been applied in this field, including the successful U-Net. In this work, we firstly modify the U-Net with functional blocks aiming to pursue higher performance. The absence of the expected performance boost then lead us to dig into the opposite direction of shrinking the U-Net and exploring the extreme conditions such that its segmentation performance is maintained. Experiment series to simplify the network structure, reduce the network size and restrict the training conditions are designed. Results show that for retinal vessel segmentation on DRIVE database, U-Net does not degenerate until surprisingly acute conditions: one level, one filter in convolutional layers, and one training sample. This experimental discovery is both counter-intuitive and worthwhile. Not only are the extremes of the U-Net explored on a well-studied application, but also one intriguing warning is raised for the research methodology which seeks for marginal performance enhancement regardless of the resource cost.
CRIS-Team Scopus Importer:2020-04-21

Form Follows Function: Smart Network Design Enables Zero-shot Network Reuse - Fu, Weilin; Husvogt, Lennart; Breininger, Katharina; Schaffert, Roman; Abu-Qamar, Omar; Fujimoto, James G.; Maier, Andreas (Informatik aktuell, 2022)
In this work, we construct and train a learning-based neural network pipeline for retinal vessel segmentation on fundus images and transfer the preprocessing module directly onto OCTA en face projections. Without additional fine-tuning, the transferred module retains the edge-preserving denoising functionality, as our smart network design enables zero-shot domain adaptation. Compared to a denoising network trained on OCTA data, the transferred preprocessing module is superior with regard to performance, generalization ability, and numerical stability. In addition, Frangi’s vessel segmentation of the preprocessed images outperforms the predictions using pretrained full segmentation networks. The selection of fitting Gaussian scales of the Frangi filter in the target domain can bypass the over/under-segmentation problem caused by the vessel diameter mismatch between the two domains. The contribution of this work is two-fold: we provide an exemplary evidence that smart design of network pipelines allows for flexible functional module reuse across different imaging modalities, i.e. zero-shot domain adaptation; and we discover a deep learning-based vessel segmentation pipeline for OCTA en face images which reaches an AUC score of 0.9418 without additional training.<br
CRIS-Team Scopus Importer:2022-05-06

Frangi-Net: A Neural Network Approach to Vessel Segmentation - Fu, Weilin; Breininger, Katharina; Schaffert, Roman; Ravikumar, Nishant; Würfl, Tobias; Fujimoto, James G.; Moult, Eric M.; Maier, Andreas (BildVerarbeitung für die Medizin (BVM) 2018, 2018)
In this paper, we reformulate the conventional 2-D Frangi vesselness measure into a pre-weighted neural network (“Frangi-Net”), and illustrate that the Frangi-Net is equivalent to the original Frangi filter. Furthermore, we show that, as a neural network, Frangi-Net is trainable. We evaluate the proposed method on a set of 45 high resolution fundus images. After fine-tuning, we observe both qualitative and quantitative improvements in the segmentation quality compared to the original Frangi measure, with an increase up to 17% in F1 scor
Keywords: Vessel segmentation; Precision learning; Frangi-Net; Fundus imaging

Imitation learning network for fundus image registration using a divide-and-conquer approach - Bayer, Siming; Zhong, Xia; Fu, Weilin; Ravikumar, Nishant; Maier, Andreas (Informatik aktuell, 2020)
Comparison of microvascular circulation on fundoscopic images is a non-invasive clinical indication for the diagnosis and monitoring of diseases, such as diabetes and hypertensions. The differences between intra-patient images can be assessed quantitatively by registering serial acquisitions. Due to the variability of the images (i.e. contrast, luminosity) and the anatomical changes of the retina, the registration of fundus images remains a challenging task. Recently, several deep learning approaches have been proposed to register fundus images in an end-to-end fashion, achieving remarkable results. However, the results are diffcult to interpret and analyze. In this work, we propose an imitation learning framework for the registration of 2D color funduscopic images for a wide range of applications such as disease monitoring, image stitching and super-resolution. We follow a divide-and-conquer approach to improve the interpretability of the proposed network, and analyze both the influence of the input image and the hyperparameters on the registration result. The results show that the proposed registration network reduces the initial target registration error up to 95%.
CRIS-Team Scopus Importer:2020-04-21

“Keep it simple, scholar”: an experimental analysis of few-parameter segmentation networks for retinal vessels in fundus imaging - Fu, Weilin; Breininger, Katharina; Schaffert, Roman; Pan, Zhaoya; Maier, Andreas (International Journal of Computer Assisted Radiology and Surgery, 2021)
Purpose: With the recent development of deep learning technologies, various neural networks have been proposed for fundus retinal vessel segmentation. Among them, the U-Net is regarded as one of the most successful architectures. In this work, we start with simplification of the U-Net, and explore the performance of few-parameter networks on this task. Methods: We firstly modify the model with popular functional blocks and additional resolution levels, then we switch to exploring the limits for compression of the network architecture. Experiments are designed to simplify the network structure, decrease the number of trainable parameters, and reduce the amount of training data. Performance evaluation is carried out on four public databases, namely DRIVE, STARE, HRF and CHASE_DB1. In addition, the generalization ability of the few-parameter networks are compared against the state-of-the-art segmentation network. Results: We demonstrate that the additive variants do not significantly improve the segmentation performance. The performance of the models are not severely harmed unless they are harshly degenerated: one level, or one filter in the input convolutional layer, or trained with one image. We also demonstrate that few-parameter networks have strong generalization ability. Conclusion: It is counter-intuitive that the U-Net produces reasonably good segmentation predictions until reaching the mentioned limits. Our work has two main contributions. On the one hand, the importance of different elements of the U-Net is evaluated, and the minimal U-Net which is capable of the task is presented. On the other hand, our work demonstrates that retinal vessel segmentation can be tackled by surprisingly simple configurations of U-Net reaching almost state-of-the-art performance. We also show that the simple configurations have better generalization ability than state-of-the-art models with high model complexity. These observations seem to be in contradiction to the current trend of continued increase in model complexity and capacity for the task under consideration.
CRIS-Team Scopus Importer:2021-05-21
Keywords: Computation cost; Fundus image; U-Net; Vessel segmentation

Learning with Known Operators reduces Maximum Training Error Bounds. - Maier, Andreas; Syben-Leisner, Christopher; Stimpel, Bernhard; Würfl, Tobias; Hoffmann, Mathis; Schebesch, Frank; Fu, Weilin; Mill, Leonid; Kling, Lasse; Christiansen, Silke H. (Nature Machine Intelligence, 2019)
We describe an approach for incorporating prior knowledge into machine learning algorithms. We aim at applications in physics and signal processing in which we know that certain operations must be embedded into the algorithm. Any operation that allows computation of a gradient or sub-gradient towards its inputs is suited for our framework. We derive a maximal error bound for deep nets that demonstrates that inclusion of prior knowledge results in its reduction. Furthermore, we also show experimentally that known operators reduce the number of free parameters. We apply this approach to various tasks ranging from CT image reconstruction over vessel segmentation to the derivation of previously unknown imaging algorithms. As such the concept is widely applicable for many researchers in physics, imaging, and signal processing. We assume that our analysis will support further investigation of known operators in other fields of physics, imaging, and signal processing.

Modularization of deep networks allows cross-modality reuse: lesson learnt - Fu, Weilin; Husvogt, Lennart; Ploner, Stefan; Fujimoto, James G.; Maier, Andreas (Informatik aktuell, 2020)
Fundus photography and Optical Coherence Tomography Angiography (OCT-A) are two commonly used modalities in ophthalmic imaging. With the development of deep learning algorithms, fundus image processing, especially retinal vessel segmentation, has been extensively studied. Built upon the known operator theory, interpretable deep network pipelines with well-defined modules have been constructed on fundus images. In this work, we firstly train a modularized network pipeline for the task of retinal vessel segmentation on the fundus database DRIVE. The pretrained preprocessing module from the pipeline is then directly transferred onto OCT-A data for image quality enhancement without further fine-tuning. Output images show that the preprocessing net can balance the contrast, suppress noise and thereby produce vessel trees with improved connectivity in both image modalities. The visual impression is confirmed by an observer study with five OCT-A experts. Statistics of the grades by the experts indicate that the transferred module improves both the image quality and the diagnostic quality. Our work provides an example that modules within network pipelines that are built upon the known operator theory facilitate cross-modality reuse without additional training or transfer learning.
CRIS-Team Scopus Importer:2020-04-21

Novel Evaluation Metrics for Vascular Structure Segmentation - Reimann, Marcel; Fu, Weilin; Maier, Andreas (Informatik aktuell, 2021)
CRIS-Team Scopus Importer:2021-04-09

Odin’s Eye – A Close Look at Gamification of Labelling of Ophthalmic Diseases - Fan, Fuxin; Qiu, Jingna; Wang, JiaWei; Valianos, Stelica; Farooq, Muhammad Muqtasid; Fu, Weilin; Kordon, Florian; Maier, Andreas (Joint Conference of the GMDS & CEN-IBS 2020, 2020)
Introduction State of the art deep learning [1] requires vast amounts of accurately labeled training data to enable high classification performance [2]. In order to obtain sufficient amounts of data, data donation is a feasible approach [3]. Yet, the data is only usable, if correct annotations are present. One way to create such annotations is crowd-sourcing via gamification. In this paper, we present an integrated training and annotation approach that allows large scale annotation of ophthalmic diseases. Methods We created a game called “Odin’s Eye” in order to make image classification an exciting and rewarding experience. The game has three modes that are used to slowly lead the player to the complex field of ophthalmic diseases. In order to do so, we used image data from the ODIR 2019 Challenge (https://odir2019.grand-challenge.org/dataset/). The dataset contains more than 7000 images showing healthy eye data and different pathologies. In “Normal Mode”, the player is shown one fundus image and is asked to select whether the shown image is “normal” or shows Cataract, Macular Degeneration, Glaucoma, Retinopathy, or Myopia. After a sufficient number of correct answers, the player can access the unlabeled mode, in which the user is able to give annotation to unlabeled image. The annotations for the unlabeled images will be shown in the “Unlabeled Result”. The “Difficult Mode” is inspired by puzzle games such as Candy Crush or Zookeeper. Here, the player is displayed various fundus images and is asked to align them such that three images showing the same pathology form either a row or a column. Once such a triplet is found, it is eliminated and new images enter the game canvas from the top. In order to make the game more challenging, a curtain enters the field of view from the top that gradually increases the pressure on the player as well as increases game difficulty via decreasing the image brightness. After each successful elimination, the curtain is raised a little and the time limit is increased. Rewarding strategies like reward animation and image shuffling are applied to achieve better user experience. The idea of using triplets is appealing as images of unknown pathology can be mixed in the game area. The player will implicitly classify such images when he tries to align them with two more images of the same class. A leader board shows high scores in order to encourage players to compare their scores online and to compete in creating a large number of annotations. Results A prototype was implemented in Unity [4]. In order to test the game idea, only five prototypical images were chosen for each pathology at present. The difficulty of the game can easily be increased by increasing types of diseases. First experiences with test players confirmed that in particular the “Difficult Mode” has a very rewarding game experience. Prototypes are available as Android APK and WebGL (https://www.medicaldatadonors.org/index.php/odins-eye/). An in-game video was created to demonstrate the gameplay (https://youtu.be/UehnND9gkvY). Conclusion A believe that we created a challenging yet rewarding game experience. In future versions, we will make use of additional images from the ODIR 2019 dataset in order to create slowly increasing difficulty in the game to convince even expert players to keep playing Odin’s Eye. References [1] Maier, A., Syben, C., Lasser, T., & Riess, C. (2019). A gentle introduction to deep learning in medical image processing. Zeitschrift für Medizinische Physik, 29(2), 86-101. [2] Bertram, C. A., Aubreville, M., Marzahl, C., Maier, A., & Klopfleisch, R. (2019). A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Scientific data, 6(1), 1-9. [3] Servadei, L., Schmidt, R., Eidelloth, C., & Maier, A. (2017, October). Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems" (pp. 87-97). Springer, Cham. [4] Murray, J. W. (2014). C# game programming cookbook for Unity 3D. AK Peters/CRC Press
Keywords: Gamification; Medical Image Annotation; Deep Learning

ScanRacer – Fast Track Annotation of Segmentation Masks in Volumetric Medical Images - Diehm, Jessica; Hermann, Robert; Pertlwieser, Tobias; Cheng, Wei; Fu, Weilin; Kordon, Florian; Maier, Andreas (HEALTHINF - 14th International Conference on Health Informatics, 2021)
Introduction With the rise of deep learning [1], we see a dramatic need of curated and annotated medical image data. In particular, in volumetric images, such annotations are extremely costly, as each slice has to be outlined individually to generate ground truth for the training of deep learning algorithms. Recently, a new initiative was founded that aims at encouraging patients to donate their medical image data [2]. In particular, this initiative also asks for permission to crowd-source the data annotation. This forms a basis to generate sufficient data for large-scale training of deep learning algorithms in medical image analysis. Methods In order to encourage users to perform annotations, we explore gamification. In particular, we selected the setup of a racing game to generate an exciting user experience. The main idea is that the user is driving a race car across a volumetric image slice. In order to create tracks automatically, a simple segmentation method is required. For the first experiments, we chose thresholding to separate fore- and background. Then, we select the largest connected component and perform image processing to automatically extract a closed edge contour. As guidance, checkpoints are spread over the preliminary organ outline to guide the player in equidistant steps along the extracted contour. Based on edge detection [3], a score consisting of accumulated edge pixels is determined automatically as feedback for the player. In order to create a more challenging game experience, additional moving obstacles were added to the course. During driving the player creates a closed contour that is then transmitted from the game client to the server’s database. Results The game was implemented in Unity3D [4] and released as Android APK Installer (https://www.medicaldatadonors.org/index.php/scan-racer/). We chose Google Firebase to store segmentation results on the server (https://firebase.google.com). ScanRacer creates a challenging, yet rewarding experience for the user. Experienced players are able to create segmentation contours close to the correct segmentation outline. Yet detailed annotations still pose a challenge in this setup which will be addressed by the addition of further game elements. In contrast to many other serious games, the fun component of ScanRacer is very high as reported by test players. A gameplay teaser video demonstrates this in detail. (https://www.youtube.com/watch?v=JNmEGLCyf6w) Conclusion ScanRacer offers a unique experience for players for volumetric image segmentation. The game is was implemented in Unity3D and is available for free download. Sources can be shared at request. References [1] Maier, A., Syben, C., Lasser, T., & Riess, C. (2019). A gentle introduction to deep learning in medical image processing. Zeitschrift für Medizinische Physik, 29(2), 86-101. [2] Servadei, L., Schmidt, R., Eidelloth, C., & Maier, A. (2017, October). Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems" (pp. 87-97). Springer, Cham. [3] Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6), 679-698. [4] Murray, J. W. (2014). C# game programming cookbook for Unity 3D. AK Peters/CRC Press<br
Keywords: Gamification; Medical Image Annotation; Deep Learning

Towards Mouse Bone X-ray Microscopy Scan Simulation - Fu, Weilin; Mill, Leonid; Seitz, Stephan; Geimer, Tobias; Kling, Lasse; Possart, Dennis; Christiansen, Silke; Maier, Andreas (Informatik aktuell, 2021)
CRIS-Team Scopus Importer:2021-04-19

Velo-CT – A Collaborative 3D Racing Game as Incentive for Medical Image Annotation - Sukesh, Richin; Goshika, Sai Kishore; Narayanan, Srividhya Sathya; Bhombore, Vineet Vinay; Fu, Weilin; Kordon, Florian; Maier, Andreas (HEALTHINF - 14th International Conference on Health Informatics, 2021)
Introduction Recent advances in algorithmic processing of medical data are largely based on the use of deep learning algorithms that rely on large amounts of annotated data [1]. To cope with the often complex and time-consuming nature of data acquisition and annotation, data donation and distributed crowd-labeling are an appealing approach [2]. In this paper, we follow this idea and present a single/multi-player mobile game which aids in the annotation process of CT or MR scan data. Methods To provide users with an exciting and rewarding annotation experience, our game "Velo-CT" integrates labeling of 3D volume slices into a classical racing game. This is achieved by using data annotation as a key to unlock more advanced racing tracks in the game. To this end, the game has two game modes: 1) Track Racing, and 2) Image Annotation. In “Track Racing”, players compete by steering a vehicle through a racetrack. The main task is to overtake the opponents and try to remain first in position until the end of the race. Checkpoints along the track allow additional guidance if players accidentally leave the racetrack. To provide a more challenging and diverse experience, respawning pick-ups can be found at random locations along the track in the form of floating CT scanners. Such pick-up can either be a speed multiplier (multiplies car speed by 1.2) or a size enhancer (increases the car size by 1.5 to threat opponents and impede their view). The racetracks are modeled based on the segmentation contour of clinical CT data, e.g. showing the respiratory region. For slice segmentation we use a simple K-means classifier. Annotation points earned after beating a track allow the player to unlock new data for the second game mode “Image Annotation”. This is where the medical expertise of the player comes into play. Based on the current progress, the player is presented several medical images which may contain abnormalities. If an abnormality is spotted, the precise location can be marked by a finger tap on the area. The more images a player annotates, the more points can be earned to unlock a new track – and eventually a more intricate anatomy to race on. Results Our game was prototyped in Unity3D [3]. As a first anatomy, respiratory tracks based on the “Low Dose CT Grand Challenge” data were constructed. First experiences by players of different gaming background were reported for the single player setting. Users report a fast-paced racing experience with a healthy amount of challenge provided by pick-ups and competitive AI opponents. The organ-aligned layouts result in tracks similar in shape, which should be addressed in the future by adding multiple anatomical regions. Conclusion Velo-CT provides an intuitive and extendable integration of the often tedious data annotation process for 3D medical data into a rewarding gaming context. The prototype is available for free download and serves as a basis for further extensions to related game modes. References [1] Maier, A., Syben, C., Lasser, T., & Riess, C. (2019). A gentle introduction to deep learning in medical image processing. Zeitschrift für Medizinische Physik, 29(2), 86-101. [2] Servadei, L., Schmidt, R., Eidelloth, C., & Maier, A. (2017, October). Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems" (pp. 87-97). Springer, Cham. [3] Murray, J. W. (2014). C# game programming cookbook for Unity 3D. AK Peters/CRC Press. <br
Keywords: Gamification; Medical Image Annotation; Deep Learning.