Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain MRI

Zeineldin R, Karar ME, Elshaer Z, Coburger J, Wirtz CR, Burgert O, Mathis-Ullrich F (2024)

Publication Type: Journal article

Publication year: 2024

Journal

Scientific Reports Nature Publishing Group: Open Access Journals - Option B

Book Volume: 14

Article Number: 3713

Journal Issue: 1

DOI: 10.1038/s41598-024-54186-7

Abstract

Accurate localization of gliomas, the most common malignant primary brain cancer, and its different sub-region from multimodal magnetic resonance imaging (MRI) volumes are highly important for interventional procedures. Recently, deep learning models have been applied widely to assist automatic lesion segmentation tasks for neurosurgical interventions. However, these models are often complex and represented as “black box” models which limit their applicability in clinical practice. This article introduces new hybrid vision Transformers and convolutional neural networks for accurate and robust glioma segmentation in Brain MRI scans. Our proposed method, TransXAI, provides surgeon-understandable heatmaps to make the neural networks transparent. TransXAI employs a post-hoc explanation technique that provides visual interpretation after the brain tumor localization is made without any network architecture modifications or accuracy tradeoffs. Our experimental findings showed that TransXAI achieves competitive performance in extracting both local and global contexts in addition to generating explainable saliency maps to help understand the prediction of the deep network. Further, visualization maps are obtained to realize the flow of information in the internal layers of the encoder-decoder network and understand the contribution of MRI modalities in the final prediction. The explainability process could provide medical professionals with additional information about the tumor segmentation results and therefore aid in understanding how the deep learning model is capable of processing MRI data successfully. Thus, it enables the physicians’ trust in such deep learning systems towards applying them clinically. To facilitate TransXAI model development and results reproducibility, we will share the source code and the pre-trained models after acceptance at https://github.com/razeineldin/TransXAI.

Authors with CRIS profile

Ramy Zeineldin Professur für Robotische Planung und Kognition in der Chirurgie Franziska Mathis-Ullrich Professur für Robotische Planung und Kognition in der Chirurgie

Involved external institutions

Menoufia University (Minoufiya / Minufiya University) / جامعة المنوفية‎ EG Egypt (EG) Universität Ulm DE Germany (DE) Hochschule Reutlingen DE Germany (DE)

How to cite

APA:

Zeineldin, R., Karar, M.E., Elshaer, Z., Coburger, J., Wirtz, C.R., Burgert, O., & Mathis-Ullrich, F. (2024). Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain MRI. Scientific Reports, 14(1). https://doi.org/10.1038/s41598-024-54186-7

MLA:

Zeineldin, Ramy, et al. "Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain MRI." Scientific Reports 14.1 (2024).

BibTeX: Download