An Experimental Survey of Incremental Transfer Learning for Multicenter Collaboration - Huang, Yixing; Bert, Christoph; Gomaa, Ahmed; Fietkau, Rainer; Maier, Andreas; Putz, Florian (IEEE Access, 2024)
Keywords: Continual learning; multicenter collaboration; federated learning; data privacy; deep learning; peer-to-peer; domain incremental learning; incremental transfer learning

Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for AI-assisted medical education and decision making in radiation oncology - Huang, Yixing; Gomaa, Ahmed; Semrau, Sabine; Haderlein, Marlen; Lettmaier, Sebastian; Weissmann, Thomas; Grigo, Johanna; Ben Tkhayat, Hassen; Frey, Benjamin; Gaipl, Udo; Distel, Luitpold; Maier, Andreas; et al. (Frontiers in Oncology, 2023)
Keywords: Large language model; Radiotherapy; Natural language processing; Artificial intelligence; Gray Zone; Clinical decision support (CDS)

Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma - Gomaa, Ahmed; Huang, Yixing; Hagag, Amr; Schmitter, Charlotte; Höfler, Daniel; Weissmann, Thomas; Breininger, Katharina; Schmidt, Manuel; Stritzelberger, Jenny; Delev, Daniel; Coras, Roland; Dörfler, Arnd; et al. (Neuro-Oncology Advances, 2024)
Keywords: Survival prediction; deep learning; glioblastoma; MR; prognosis; multimodal data

Deep Learning for Cancer Prognosis Prediction Using Portrait Photos by StyleGAN Embedding - Hagag, Amr; Gomaa, Ahmed; Kornek, Dominik; Maier, Andreas; Fietkau, Rainer; Bert, Christoph; Huang, Yixing; Putz, Florian (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Survival prediction for cancer patients is critical for optimal treatment selection and patient management. Current patient survival prediction methods typically extract survival information from patients’ clinical record data or biological and imaging data. In practice, experienced clinicians can have a preliminary assessment of patients’ health status based on patients’ observable physical appearances, which are mainly facial features. However, such assessment is highly subjective. In this work, the efficacy of objectively capturing and using prognostic information contained in conventional portrait photographs using deep learning for survival prediction purposes is investigated for the first time. A pre-trained StyleGAN2 model is fine-tuned on a custom dataset of our cancer patients’ photos to empower its generator with generative ability suitable for patients’ photos. The StyleGAN2 is then used to embed the photographs to its highly expressive latent space. Utilizing state-of-the-art survival analysis models and StyleGAN’s latent space embeddings, this approach predicts the overall survival for single as well as pan-cancer, achieving a C-index of 0.680 in a pan-cancer analysis, showcasing the prognostic value embedded in simple 2D facial images. In addition, thanks to StyleGAN’s interpretable latent space, our survival prediction model can be validated for relying on essential facial features, eliminating any biases from extraneous information like clothing or background. Moreover, our approach provides a novel health attribute obtained from StyleGAN’s extracted features, allowing the modification of face photographs to either a healthier or more severe illness appearance, which has significant prognostic value for patient care and societal perception, underscoring its potential important clinical value.
CRIS-Team Scopus Importer:2024-10-25
Keywords: cancer; deep learning; explainable AI; face prognosis; latent space; StyleGAN; Survival prediction

Multicenter privacy-preserving model training for deep learning brain metastases autosegmentation - Huang, Yixing; Khodabakhshi, Zahra; Gomaa, Ahmed; Schmidt, Manuel; Fietkau, Rainer; Guckenberger, Matthias; Andratschke, Nicolaus; Bert, Christoph; Tanadini-Lang, Stephanie; Putz, Florian (Radiotherapy and Oncology, 2024)
OBJECTIVES: This work aims to explore the impact of multicenter data heterogeneity on deep learning brain metastases (BM) autosegmentation performance, and assess the efficacy of an incremental transfer learning technique, namely learning without forgetting (LWF), to improve model generalizability without sharing raw data. MATERIALS AND METHODS: A total of six BM datasets from University Hospital Erlangen (UKER), University Hospital Zurich (USZ), Stanford, UCSF, New York University (NYU), and BraTS Challenge 2023 were used. First, the performance of the DeepMedic network for BM autosegmentation was established for exclusive single-center training and mixed multicenter training, respectively. Subsequently privacy-preserving bilateral collaboration was evaluated, where a pretrained model is shared to another center for further training using transfer learning (TL) either with or without LWF. RESULTS: For single-center training, average F1 scores of BM detection range from 0.625 (NYU) to 0.876 (UKER) on respective single-center test data. Mixed multicenter training notably improves F1 scores at Stanford and NYU, with negligible improvement at other centers. When the UKER pretrained model is applied to USZ, LWF achieves a higher average F1 score (0.839) than naive TL (0.570) and single-center training (0.688) on combined UKER and USZ test data. Naive TL improves sensitivity and contouring accuracy, but compromises precision. Conversely, LWF demonstrates commendable sensitivity, precision and contouring accuracy. When applied to Stanford, similar performance was observed. CONCLUSION: Data heterogeneity (e.g., variations in metastases density, spatial distribution, and image spatial resolution across centers) results in varying performance in BM autosegmentation, posing challenges to model generalizability. LWF is a promising approach to peer-to-peer privacy-preserving model training.
EVALuna2:587588
Keywords: Continual learning; Learning without forgetting; Brain metastasis; Multicenter collaboration; Data privacy; Data heterogeneity

Principles of artificial intelligence in radiooncology - Huang, Yixing; Gomaa, Ahmed; Höfler, Daniel; Schubert, Philipp; Gaipl, Udo; Frey, Benjamin; Fietkau, Rainer; Bert, Christoph; Putz, Florian (Strahlentherapie und Onkologie, 2024)
Keywords: Artificial intelligence; Radiation Oncology; Deep learning

Supervised Contrastive Learning for Robust and Efficient Multi-modal Emotion and Sentiment Analysis - Gomaa, Ahmed; Maier, Andreas; Kosti, Ronak Vijaypal (2022 26TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2022)
CRIS-Team WoS Importer:2023-02-17

Task-Specific Data Preparation for Deep Learning to Reconstruct Structures of Interest from Severely Truncated CBCT Data - Huang, Yixing; Fan, Fuxin; Gomaa, Ahmed; Maier, Andreas; Fietkau, Rainer; Bert, Christoph; Putz, Florian (The 8th International Conference on Image Formation in X-Ray Computed Tomography, 2024)
Cone-beam computed tomography (CBCT) is widely used in interventional surgeries and radiation oncology. Due to the limited size of flat-panel detectors, anatomical structures might be missing outside the limited field-of-view (FOV), which restricts the clinical applications of CBCT systems. Recently, deep learning methods have been proposed to extend the FOV for multi-slice CT systems. However, in mobile CBCT system with a smaller FOV size, projection data is severely truncated and it is challenging for a network to restore all missing structures outside the FOV. In some applications, only certain structures outside the FOV are of interest, e.g., ribs in needle path planning for liver/lung cancer diagnosis. Therefore, a task-specific data preparation method is proposed in this work, which automatically let the network focus on structures of interest instead of all the structures. Our preliminary experiment shows that Pix2pixGAN with a conventional training has the risk to reconstruct false positive and false negative rib structures from severely truncated CBCT data, whereas Pix2pixGAN with the proposed task-specific training can reconstruct all the ribs reliably. The proposed method is promising to empower CBCT with more clinical applications.<br
Keywords: Field-of-view extension; computed tomography; deep learning; data truncation; task-specific learning.

The Segment Anything foundation model achieves favorable brain tumor auto-segmentation accuracy in MRI to support radiotherapy treatment planning - Putz, Florian; Beirami, Sogand; Schmidt, Manuel; May, Matthias; Grigo, Johanna; Weissmann, Thomas; Schubert, Philipp; Höfler, Daniel; Gomaa, Ahmed; Hassen, Ben Tkhayat; Lettmaier, Sebastian; Frey, Benjamin; et al. (Strahlentherapie und Onkologie, 2024)
Background: Promptable foundation auto-segmentation models like Segment Anything (SA, Meta AI, New York, USA) represent a novel class of universal deep learning auto-segmentation models that could be employed for interactive tumor auto-contouring in RT treatment planning. Methods: Segment Anything was evaluated in an interactive point-to-mask auto-segmentation task for glioma brain tumor auto-contouring in 16,744 transverse slices from 369 MRI datasets (BraTS 2020 dataset). Up to nine interactive point prompts were automatically placed per slice. Tumor boundaries were auto-segmented on contrast-enhanced T1w sequences. Out of the three auto-contours predicted by SA, accuracy was evaluated for the contour with the highest calculated IoU (Intersection over Union, “oracle mask,” simulating interactive model use with selection of the best tumor contour) and for the tumor contour with the highest model confidence (“suggested mask”). Results: Mean best IoU (mbIoU) using the best predicted tumor contour (oracle mask) in full MRI slices was 0.762 (IQR 0.713–0.917). The best 2D mask was achieved after a mean of 6.6 interactive point prompts (IQR 5–9). Segmentation accuracy was significantly better for high- compared to low-grade glioma cases (mbIoU 0.789 vs. 0.668). Accuracy was worse using the suggested mask (0.572). Stacking best tumor segmentations from transverse MRI slices, mean 3D Dice score for tumor auto-contouring was 0.872, which was improved to 0.919 by combining axial, sagittal, and coronal contours. Conclusion: The Segment Anything foundation segmentation model can achieve high accuracy for glioma brain tumor segmentation in MRI datasets. The results suggest that foundation segmentation models could facilitate RT treatment planning when properly integrated in a clinical application.
CRIS-Team Scopus Importer:2024-11-15
Keywords: Artificial intelligence; Deep learning; Glioma; Interactive segmentation; Tumor auto-contouring