Abstract: Leveraging Image Captions for Selective Whole Slide Image Annotation - Qiu, Jingna; Aubreville, Marc; Wilm, Frauke; Öttl, Mathias; Utz, Jonas; Schlereth, Maja; Breininger, Katharina (Bildverarbeitung für die Medizin 2025, 2025)
Obtaining dense annotations for histopathological whole-slide images (WSI), such as segmentation masks or mitotic figure identification, is a labor intensive process due to the large image size and the extensive manual effort required for annotation. Identifying informative regions in WSIs for annotation while leaving other regions unlabeled can significantly reduce the annotation effort. These selected annotation regions should contain valuable training information that allows proper model training without significantly impacting performance compared to full annotatio

Abstract: Pan-tumor CAnine CuTaneous Cancer Histology (CATCH) Dataset - Wilm, Frauke; Fragoso, Marco; Marzahl, Christian; Qiu, Jingna; Puget, Chloé; Diehl, Laura; Bertram, Christof A.; Klopfleisch, Robert; Maier, Andreas; Breininger, Katharina; Aubreville, Marc (Informatik aktuell, 2023)
The identification of tumor regions on cutaneous tissue sections and the subsequent differentiation into individual tumor subtypes are routine tasks for veterinary pathologists. However, manual tumor delineation can be time-consuming and morphological similarities of tumor types can make the subtyping task difficult. Deep learning-based algorithms have been successfully applied for both tasks but require a large amount of annotated data to learn robust representations. Acquiring the annotations necessary to train these algorithms can be time-consuming and requires expert knowledge. Publicly available datasets reduce this annotation overhead and allow for a comparison of image analysis algorithms trained for the task at hand. We published a publicly available dataset of 350 whole slide images (WSIs) of seven different canine cutaneous tumors. These were annotated for the tumor subtypes and additionally six skin tissue classes. Overall, the database includes 12,424 polygon annotations for 13 classes, which exceeds most publicly available datasets in annotation extent and label diversity. In our dataset report, we validated the provided annotations through inter-rater experiments on a subset of the presented dataset, where three pathologists demonstrated high consistency in labeling tissue structures, especially for tumor annotations. Furthermore, we performed a technical validation by first training a deep neural network for tissue phenotyping. The model segmented the WSI into tumor and five skin tissue classes, where we achieved a class-averaged Jaccard coefficient of 0.7047 and a Jaccard coefficient of 0.9044 for tumor in particular. Afterward, we classified the segmented tumor regions into the tumor subtypes and achieved a slide-level accuracy of 0.9857. These baseline results provide a starting point for the development of more advanced algorithms. Previous works have shown that canine and human cutaneous tumors share various histologic patterns, which extends the added value of the presented dataset beyond veterinary pathology [1].
Created from Fastlane, Scopus look-up

Adaptive Region Selection for Active Learning in Whole Slide Image Semantic Segmentation - Qiu, Jingna; Wilm, Frauke; Öttl, Mathias; Schlereth, Maja; Liu, Chang; Heimann, Tobias; Aubreville, Marc; Breininger, Katharina (The 26th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023), 2023)
The process of annotating histological gigapixel-sized whole slide images (WSIs) at the pixel level for the purpose of training a supervised segmentation model is time-consuming. Region-based active learning (AL) involves training the model on a limited number of annotated image regions instead of requesting annotations of the entire images. These annotation regions are iteratively selected, with the goal of optimizing model performance while minimizing the annotated area. The standard method for region selection evaluates the informativeness of all square regions of a specified size and then selects a specific quantity of the most informative regions. We find that the efficiency of this method highly depends on the choice of AL step size (i.e., the combination of region size and the number of selected regions per WSI), and a suboptimal AL step size can result in redundant annotation requests or inflated computation costs. This paper introduces a novel technique for selecting annotation regions adaptively, mitigating the reliance on this AL hyperparameter. Specifically, we dynamically determine each region by first identifying an informative area and then detecting its optimal bounding box, as opposed to selecting regions of a uniform predefined shape and size as in the standard method. We evaluate our method using the task of breast cancer metastases segmentation on the public CAMELYON16 dataset and show that it consistently achieves higher sampling efficiency than the standard method across various AL step sizes. With only 2.6% of tissue area annotated, we achieve full annotation performance and thereby substantially reduce the costs of annotating a WSI dataset. The source code is available at https://github.com/DeepMicroscopy/AdaptiveRegionSelection.
Keywords: Active learning; Region selection; Whole slide images.

Deep learning-based classification of erosion, synovitis and osteitis in hand MRI of patients with inflammatory arthritis - Schlereth, Maja; Yalcin-Mutlu, Melek; Utz, Jonas; Bayat, Sara; Heimann, Tobias; Qiu, Jingna; Ehring, Chris; Liu, Chang; Uder, Michael; Kleyer, Arnd; Simon, David; Roemer, Frank; et al. (RMD Open, 2024)
OBJECTIVES: To train, test and validate the performance of a convolutional neural network (CNN)-based approach for the automated assessment of bone erosions, osteitis and synovitis in hand MRI of patients with inflammatory arthritis. METHODS: Hand MRIs (coronal T1-weighted, T2-weighted fat-suppressed, T1-weighted fat-suppressed contrast-enhanced) of rheumatoid arthritis (RA) and psoriatic arthritis (PsA) patients from the rheumatology department of the Erlangen University Hospital were assessed by two expert rheumatologists using the Outcome Measures in Rheumatology-validated RA MRI Scoring System and PsA MRI Scoring System scores and were used to train, validate and test CNNs to automatically score erosions, osteitis and synovitis. Scoring performance was compared with human annotations in terms of macro-area under the receiver operating characteristic curve (AUC) and balanced accuracy using fivefold cross-validation. Validation was performed on an independent dataset of MRIs from a second patient cohort. RESULTS: In total, 211 MRIs from 112 patients (14 906 region of interests (ROIs)) were included for training/internal validation using cross-validation and 220 MRIs from 75 patients (11 040 ROIs) for external validation of the networks. The networks achieved high mean (SD) macro-AUC of 92%±1% for erosions, 91%±2% for osteitis and 85%±2% for synovitis. Compared with human annotation, CNNs achieved a high mean Spearman correlation for erosions (90±2%), osteitis (78±8%) and synovitis (69±7%), which remained consistent in the validation dataset. CONCLUSIONS: We developed a CNN-based automated scoring system that allowed a rapid grading of erosions, osteitis and synovitis with good diagnostic accuracy and using less MRI sequences compared with conventional scoring. This CNN-based approach may help develop standardised cost-efficient and time-efficient assessments of hand MRIs for patients with arthritis.
CRIS-Team Scopus Importer:2024-06-28
Keywords: classification; machine learning; magnetic resonance imaging; psoriatic arthritis; rheumatoid arthritis

Enhancing Cross-Modality Synthesis: Subvolume Merging for MRI-to-CT Conversion - Fan, Fuxin; Qiu, Jingna; Huang, Yixing; Maier, Andreas (The 8th International Conference on Image Formation in X-Ray Computed Tomography, 2024)
Providing more precise tissue attenuation information, synthetic computed tomography (sCT) generated from magnetic resonance imaging (MRI) contributes to improved radiation therapy treatment planning. In our study, we employ the advanced SwinUNETR framework for synthesizing CT from MRI images. Additionally, we introduce a three-dimensional subvolume merging technique in the prediction process. By selecting an optimal overlap percentage for adjacent subvolumes, stitching artifacts are effectively mitigated, leading to a decrease in the mean absolute error (MAE) between sCT and the labels from 52.65 HU to 47.75 HU. Furthermore, implementing a weight function with a gamma value of 0.9 results in the lowest MAE within the same overlap area. By setting the overlap percentage between 50% and 70%, we achieve a balance between image quality and computational efficiency.<br
Keywords: CT synthesis; Swin transformer; subvolume merging

Fully automatic HER2 tissue segmentation for interpretable HER2 scoring - Öttl, Mathias; Steenpaß, Jana; Wilm, Frauke; Qiu, Jingna; Rübner, Matthias; Lang-Schwarz, Corinna; Taverna, Cecilia; Tava, Francesca; Hartmann, Arndt; Hübner, Hanna; Beckmann, Matthias; Fasching, Peter; et al. (Journal of Pathology Informatics, 2025)
Breast cancer is the most common cancer in women, with HER2 (human epidermal growth factor receptor 2) overexpression playing a critical role in regulating cell growth and division. HER2 status, assessed according to established scoring guidelines, offers important information for treatment selection. However, the complexity of the task leads to variability in human rater assessments. In this work, we propose a fully automated, interpretable HER2 scoring pipeline based on pixel-level semantic segmentations, designed to align with clinical guidelines. Using polygon annotations, our method balances annotation effort with the ability to capture fine-grained details and larger structures, such as non-invasive tumor tissue. To enhance HER2 segmentation, we propose the use of a Wasserstein Dice loss to model class relationships, ensuring robust segmentation and HER2 scoring performance. Additionally, based on observations of pathologists' behavior in clinical practice, we propose a calibration step to the scoring rules, which positively impacts the accuracy and consistency of automated HER2 scoring. Our approach achieves an F1 score of 0.832 on HER2 scoring, demonstrating its effectiveness. This work establishes a potent segmentation pipeline that can be further leveraged to analyze HER2 expression in breast cancer tissue.
CRIS-Team Scopus Importer:2025-04-11
Keywords: Deep learning; HER2; HER2 scoring; Histopathology; Semantic segmentation

Generating synthetic computed tomography for radiotherapy: SynthRAD2023 challenge report - Huijben, Evi M.C.; Terpstra, Maarten L.; Galapon, Arthur Jr; Pai, Suraj; Thummerer, Adrian; Koopmans, Peter; Afonso, Manya; van Eijnatten, Maureen; Gurney-Champion, Oliver; Chen, Zeli; Zhang, Yiwen; Zheng, Kaiyi; et al. (Medical Image Analysis, 2024)
Radiation therapy plays a crucial role in cancer treatment, necessitating precise delivery of radiation to tumors while sparing healthy tissues over multiple days. Computed tomography (CT) is integral for treatment planning, offering electron density data crucial for accurate dose calculations. However, accurately representing patient anatomy is challenging, especially in adaptive radiotherapy, where CT is not acquired daily. Magnetic resonance imaging (MRI) provides superior soft-tissue contrast. Still, it lacks electron density information, while cone beam CT (CBCT) lacks direct electron density calibration and is mainly used for patient positioning. Adopting MRI-only or CBCT-based adaptive radiotherapy eliminates the need for CT planning but presents challenges. Synthetic CT (sCT) generation techniques aim to address these challenges by using image synthesis to bridge the gap between MRI, CBCT, and CT. The SynthRAD2023 challenge was organized to compare synthetic CT generation methods using multi-center ground truth data from 1080 patients, divided into two tasks: (1) MRI-to-CT and (2) CBCT-to-CT. The evaluation included image similarity and dose-based metrics from proton and photon plans. The challenge attracted significant participation, with 617 registrations and 22/17 valid submissions for tasks 1/2. Top-performing teams achieved high structural similarity indices (≥0.87/0.90) and gamma pass rates for photon (≥98.1%/99.0%) and proton (≥97.3%/97.0%) plans. However, no significant correlation was found between image similarity metrics and dose accuracy, emphasizing the need for dose evaluation when assessing the clinical applicability of sCT. SynthRAD2023 facilitated the investigation and benchmarking of sCT generation techniques, providing insights for developing MRI-only and CBCT-based adaptive radiotherapy. It showcased the growing capacity of deep learning to produce high-quality sCT, reducing reliance on conventional CT for treatment planning.
CRIS-Team Scopus Importer:2024-08-02
Keywords: Deep learning; Medical image synthesis; Radiotherapy; Synthetic CT generation

Improved HER2 Tumor Segmentation with Subtype Balancing Using Deep Generative Networks - Öttl, Mathias; Steenpaß, Jana; Rübner, Matthias; Geppert, Carol-Immanuel; Qiu, Jingna; Wilm, Frauke; Hartmann, Arndt; Beckmann, Matthias; Fasching, Peter; Maier, Andreas; Erber, Ramona; Breininger, Katharina (Proceedings - International Symposium on Biomedical Imaging, 2023)
Tumor segmentation in histopathology images is often complicated by its composition of different histological subtypes and class imbalance. Oversampling subtypes with low prevalence features is not a satisfactory solution since it eventually leads to overfitting. We propose to create synthetic images with semantically-conditioned deep generative networks and to combine subtype-balanced synthetic images with the original dataset to achieve better segmentation performance. We show the suitability of Generative Adversarial Networks (GANs) and especially diffusion models to create realistic images based on subtype-conditioning for the use case of HER2-stained histopathology. Additionally, we show the capability of diffusion models to conditionally inpaint HER2 tumor areas with modified subtypes. Combining the original dataset with the same amount of diffusion-generated images increased the tumor Dice score from 0.833 to 0.854 and almost halved the variance between the HER2 subtype recalls. These results create the basis for more reliable automatic HER2 analysis with lower performance variance between individual HER2 subtypes.
CRIS-Team Scopus Importer:2023-10-06
Keywords: Diffusion Models; Generative Models; HER2; Histopathology; Segmentation; Subtypes

Improving Deep Learning-based Cardiac Abnormality Detection in 12-Lead ECG with Data Augmentation - Qiu, Jingna; Oppelt, Maximilian; Nissen, Michael; Anneken, Lars; Breininger, Katharina; Eskofier, Björn (: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2022)
Automated Electrocardiogram (ECG) classification using deep neural networks requires large datasets annotated by medical professionals, which is time-consuming and expensive. This work examines ECG augmentation as a method for enriching existing datasets at low cost. First, we introduce three novel augmentations: Limb Electrode Move and Chest Electrode Move both simulate a minor electrode mislocation during signal measurement, and Heart Vector Transform generates an ECG by modeling a rotated main heart axis. These techniques are then combined with nine time series signal augmentations from literature. Evaluation was performed on ICBEB, PTB-XL Diagnostic, PTB-XL Rhythm, and PTB-XL Form datasets. Compared to models trained without data augmentation, area under the receiver operating characteristic curve (AUC) was increased by 3.5%, 1.7%, 1.4% and 3.5%, respectively. Our experiments demonstrated that data augmentation can improve deep learning performance in ECG classification. Analyses of the individual augmenion effects established the efficacy of the three proposed augmentations.<br

Leveraging Image Captions for Selective Whole Slide Image Annotation - Qiu, Jingna; Aubreville, Marc; Wilm, Frauke; Öttl, Mathias; Utz, Jonas; Schlereth, Maja; Breininger, Katharina (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Acquiring annotations for whole slide images (WSIs)-based deep learning tasks, such as creating tissue segmentation masks or detecting mitotic figures, is a laborious process due to the extensive image size and the significant manual work involved in the annotation. This paper focuses on identifying and annotating specific image regions that optimize model training, given a limited annotation budget. While random sampling helps capture data variance by collecting annotation regions throughout the WSI, insufficient data curation may result in an inadequate representation of minority classes. Recent studies proposed diversity sampling to select a set of regions that maximally represent unique characteristics of the WSIs. This is done by pretraining on unlabeled data through self-supervised learning and then clustering all regions in the latent space. However, establishing the optimal number of clusters can be difficult and not all clusters are task-relevant. This paper presents prototype sampling, a new method for annotation region selection. It discovers regions exhibiting typical characteristics of each task-specific class. The process entails recognizing class prototypes from extensive histopathology image-caption databases and detecting unlabeled image regions that resemble these prototypes. Our results show that prototype sampling is more effective than random and diversity sampling in identifying annotation regions with valuable training information, resulting in improved model performance in semantic segmentation and mitotic figure detection tasks. Code is available at https://github.com/DeepMicroscopy/Prototype-sampling.
CRIS-Team Scopus Importer:2024-11-15
Keywords: annotation region selection; histopathology; image caption; low-data learning; Whole slide images

McLabel: A Local Thresholding Tool for Efficient Semi-automatic Labelling of Cells in Fluorescence Microscopy - Utz, Jonas; Schlereth, Maja; Qiu, Jingna; Thies, Mareike; Wagner, Fabian; Brahim, Oumaima B.; Gu, Mingxuan; Uderhardt, Stefan; Breininger, Katharina (Informatik aktuell, 2023)
In this work, we present a semi-automatic labelling tool for the annotation of complex cellular structures such as macrophages in fluorescence microscopy images. We present McLabel, a napari plugin that allows users to label structures of interest by simply scribbling outlines around the area of interest, using the triangle thresholding method with post-processing to identify the desired structure. Additionally, manual adaption of the threshold allows for quick and fine-grained local correction of the segmentation. The tool is evaluated in a user study with five experts, who annotated images both with and without the tool. The results show that variability in annotations between experts is reduced when the labelling tool is used and annotation time is reduced by a factor of five on average1.
CRIS-Team Scopus Importer:2023-09-29

Mind the Gap: Scanner-Induced Domain Shifts Pose Challenges for Representation Learning in Histopathology - Wilm, Frauke; Fragoso, Marco; Bertram, Christof A.; Stathonikos, Nikolas; Ottl, Mathias; Qiu, Jingna; Klopfleisch, Robert; Maier, Andreas; Aubreville, Marc; Breininger, Katharina (Proceedings - International Symposium on Biomedical Imaging, 2023)
Computer-aided systems in histopathology are often challenged by various sources of domain shift that impact the performance of these algorithms considerably. We investigated the potential of using self-supervised pre-training to overcome scanner-induced domain shifts for the downstream task of tumor segmentation. For this, we present the Barlow Triplets to learn scanner-invariant representations from a multi-scanner dataset with local image correspondences. We show that self-supervised pre-training successfully aligned different scanner representations, which, interestingly only results in a limited benefit for our downstream task. We thereby provide insights into the influence of scanner characteristics for downstream applications and contribute to a better understanding of why established self-supervised methods have not yet shown the same success on histopathology data as they have for natural images.
Created from Fastlane, Scopus look-up
Keywords: Barlow Twins; Domain Shift; Histopathology; Representation Learning

Multi-scanner Canine Cutaneous Squamous Cell Carcinoma Histopathology Dataset - Wilm, Frauke; Fragoso, Marco; Bertram, Christof A.; Stathonikos, Nikolas; Öttl, Mathias; Qiu, Jingna; Klopfleisch, Robert; Maier, Andreas; Breininger, Katharina; Aubreville, Marc (Informatik aktuell, 2023)
In histopathology, scanner-induced domain shifts are known to impede the performance of trained neural networks when tested on unseen data. Multidomain pre-training or dedicated domain-generalization techniques can help to develop domain-agnostic algorithms. For this, multi-scanner datasets with a high variety of slide scanning systems are highly desirable. We present a publicly available multi-scanner dataset of canine cutaneous squamous cell carcinoma histopathology images, composed of 44 samples digitized with five slide scanners. This dataset provides local correspondences between images and thereby isolates the scanner-induced domain shift from other inherent, e.g. morphology-induced domain shifts. To highlight scanner differences, we present a detailed evaluation of color distributions, sharpness, and contrast of the individual scanner subsets. Additionally, to quantify the inherent scanner-induced domain shift, we train a tumor segmentation network on each scanner subset and evaluate the performance both in - and cross-domain. We achieve a class-averaged in-domain intersection over union coefficient of up to 0.86 and observe a cross-domain performance decrease of up to 0.38, which confirms the inherent domain shift of the presented dataset and its negative impact on the performance of deep neural networks.
Created from Fastlane, Scopus look-up

Odin’s Eye – A Close Look at Gamification of Labelling of Ophthalmic Diseases - Fan, Fuxin; Qiu, Jingna; Wang, JiaWei; Valianos, Stelica; Farooq, Muhammad Muqtasid; Fu, Weilin; Kordon, Florian; Maier, Andreas (Joint Conference of the GMDS & CEN-IBS 2020, 2020)
Introduction State of the art deep learning [1] requires vast amounts of accurately labeled training data to enable high classification performance [2]. In order to obtain sufficient amounts of data, data donation is a feasible approach [3]. Yet, the data is only usable, if correct annotations are present. One way to create such annotations is crowd-sourcing via gamification. In this paper, we present an integrated training and annotation approach that allows large scale annotation of ophthalmic diseases. Methods We created a game called “Odin’s Eye” in order to make image classification an exciting and rewarding experience. The game has three modes that are used to slowly lead the player to the complex field of ophthalmic diseases. In order to do so, we used image data from the ODIR 2019 Challenge (https://odir2019.grand-challenge.org/dataset/). The dataset contains more than 7000 images showing healthy eye data and different pathologies. In “Normal Mode”, the player is shown one fundus image and is asked to select whether the shown image is “normal” or shows Cataract, Macular Degeneration, Glaucoma, Retinopathy, or Myopia. After a sufficient number of correct answers, the player can access the unlabeled mode, in which the user is able to give annotation to unlabeled image. The annotations for the unlabeled images will be shown in the “Unlabeled Result”. The “Difficult Mode” is inspired by puzzle games such as Candy Crush or Zookeeper. Here, the player is displayed various fundus images and is asked to align them such that three images showing the same pathology form either a row or a column. Once such a triplet is found, it is eliminated and new images enter the game canvas from the top. In order to make the game more challenging, a curtain enters the field of view from the top that gradually increases the pressure on the player as well as increases game difficulty via decreasing the image brightness. After each successful elimination, the curtain is raised a little and the time limit is increased. Rewarding strategies like reward animation and image shuffling are applied to achieve better user experience. The idea of using triplets is appealing as images of unknown pathology can be mixed in the game area. The player will implicitly classify such images when he tries to align them with two more images of the same class. A leader board shows high scores in order to encourage players to compare their scores online and to compete in creating a large number of annotations. Results A prototype was implemented in Unity [4]. In order to test the game idea, only five prototypical images were chosen for each pathology at present. The difficulty of the game can easily be increased by increasing types of diseases. First experiences with test players confirmed that in particular the “Difficult Mode” has a very rewarding game experience. Prototypes are available as Android APK and WebGL (https://www.medicaldatadonors.org/index.php/odins-eye/). An in-game video was created to demonstrate the gameplay (https://youtu.be/UehnND9gkvY). Conclusion A believe that we created a challenging yet rewarding game experience. In future versions, we will make use of additional images from the ODIR 2019 dataset in order to create slowly increasing difficulty in the game to convince even expert players to keep playing Odin’s Eye. References [1] Maier, A., Syben, C., Lasser, T., & Riess, C. (2019). A gentle introduction to deep learning in medical image processing. Zeitschrift für Medizinische Physik, 29(2), 86-101. [2] Bertram, C. A., Aubreville, M., Marzahl, C., Maier, A., & Klopfleisch, R. (2019). A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Scientific data, 6(1), 1-9. [3] Servadei, L., Schmidt, R., Eidelloth, C., & Maier, A. (2017, October). Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems" (pp. 87-97). Springer, Cham. [4] Murray, J. W. (2014). C# game programming cookbook for Unity 3D. AK Peters/CRC Press
Keywords: Gamification; Medical Image Annotation; Deep Learning

Pan-tumor CAnine cuTaneous Cancer Histology (CATCH) dataset - Wilm, Frauke; Fragoso, Marco; Marzahl, Christian; Qiu, Jingna; Puget, Chloe; Diehl, Laura; Bertram, Christof A.; Klopfleisch, Robert; Maier, Andreas; Breininger, Katharina; Aubreville, Marc (Scientific Data, 2022)
Due to morphological similarities, the differentiation of histologic sections of cutaneous tumors into individual subtypes can be challenging. Recently, deep learning-based approaches have proven their potential for supporting pathologists in this regard. However, many of these supervised algorithms require a large amount of annotated data for robust development. We present a publicly available dataset of 350 whole slide images of seven different canine cutaneous tumors complemented by 12,424 polygon annotations for 13 histologic classes, including seven cutaneous tumor subtypes. In inter-rater experiments, we show a high consistency of the provided labels, especially for tumor annotations. We further validate the dataset by training a deep neural network for the task of tissue segmentation and tumor subtype classification. We achieve a class-averaged Jaccard coefficient of 0.7047, and 0.9044 for tumor in particular. For classification, we achieve a slide-level accuracy of 0.9857. Since canine cutaneous tumors possess various histologic homologies to human tumors the added value of this dataset is not limited to veterinary pathology but extends to more general fields of application.
CRIS-Team Scopus Importer:2022-10-14

Style-Extracting Diffusion Models for Semi-supervised Histopathology Segmentation - Öttl, Mathias; Wilm, Frauke; Steenpaß, Jana; Qiu, Jingna; Rübner, Matthias; Hartmann, Arndt; Beckmann, Matthias; Fasching, Peter; Maier, Andreas; Erber, Ramona; Kainz, Bernhard; Breininger, Katharina (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2025)
Deep learning-based image generation has seen significant advancements with diffusion models, notably improving the quality of generated images. Despite these developments, generating images with unseen characteristics beneficial for downstream tasks has received limited attention. To bridge this gap, we propose Style-Extracting Diffusion Models, featuring two conditioning mechanisms. Specifically, we utilize 1) a style conditioning mechanism which allows to inject style information of previously unseen images during image generation and 2) a content conditioning which can be targeted to a downstream task, e.g., layout for segmentation. We introduce a trainable style encoder to extract style information from images, and an aggregation block that merges style information from multiple style inputs. This architecture enables the generation of images with unseen styles in a zero-shot manner, by leveraging styles from unseen images, resulting in more diverse generations. In this work, we use the image layout as target condition and first show the capability of our method on a natural image dataset as a proof-of-concept. We further demonstrate its versatility in histopathology, where we combine prior knowledge about tissue composition and unannotated data to create diverse synthetic images with known layouts. This allows us to generate additional synthetic data to train a segmentation network in a semi-supervised fashion. We verify the added value of the generated images by showing improved segmentation results and lower performance variability between patients when synthetic images are included during segmentation training. The code of the method is publicly available at https://github.com/OettlM/STEDM.
Created from Fastlane, Scopus look-up

Whole-Body Multi-Organ Segmentation Using Anatomical Attention - Liu, Chang; Denzinger, Felix; Folle, Lukas; Qiu, Jingna; Klein, Laura; Maier, Joscha; Kachelrieb, Marc; Lell, Micheal; Maier, Andreas (Proceedings - International Symposium on Biomedical Imaging, 2023)
Automated multi-organ segmentation is an increasingly important field of research enabling a holistic assessment of the human body. Deep learning-based methods, which are increasingly potent at segmentation tasks, struggle to provide anatomy consistent results due to the highly diverse input data. This e.g. may lead to the prediction of the liver in the head region as the data-driven method is not aware of typical locations of these organs. In this work, we propose to use anatomical attention to circumvent this problem. We define it as distance maps with respect to anatomical landmarks. We also introduce a pipeline to enable the use of anatomical attention. In a first step, anatomical distance maps are extracted using a lightweight attention network. These attention distance maps are then fused with the image data to provide additional anatomical information for a segmentation network. In total, the proposed pipeline is leveraged for a whole-body multi-organ segmentation task of 14 organs. In our evaluation, conducted as an ablation study, the added anatomical attention leads to an average dice score of 60.2, compared to 53.2 of an attention U-Net and 41.8 of a vanilla U-Net. Furthermore, we achieved anatomically plausible predictions.
CRIS-Team Scopus Importer:2023-10-06
Keywords: Attention; Multi-organ; Segmentation; Whole-body