Latent Space Modelling of a Biomedical Multimodal Dataset
Fischer DJ, Eckstein M, Kist A (2026)
Publication Type: Conference contribution
Publication year: 2026
Journal
Publisher: IEEE Computer Society
Book Volume: 2026-April
Conference Proceedings Title: Proceedings - International Symposium on Biomedical Imaging
Event location: London
ISBN: 9798331577636
DOI: 10.1109/ISBI61048.2026.11515907
Abstract
Clinical decision-making is rarely based on a single data source or modality. However, contemporary machine learning applications are mostly tailored to one specific modality. In this work, multimodal data integration, including histologic images and surgery-related texts, is tested for creating patient-level vector representations with information density comparable to that of pathologic analysis for Head and Neck (HNC) patients. Here, we rely on the multimodal HNCfocusing HANCOCK dataset. First, we transform individual modalities into vector representations using proven expert models tailored to their data types. Next, these are aligned into a joint latent space. Finally, the modality-specific vectors are recombined into a single patient-level embedding. With this approach, a mean AUC score of 0.67 for cancer-specific survival and 0.66 for 3 -year recurrence was achieved using only text and image data, compared to 0.79 when including structured pathologic data.
Authors with CRIS profile
Markus Eckstein
Institute of Pathology
Andreas Kist
Juniorprofessur für Artificial Intelligence in Communication Disorders
How to cite
APA:
Fischer, D.J., Eckstein, M., & Kist, A. (2026). Latent Space Modelling of a Biomedical Multimodal Dataset. In Proceedings - International Symposium on Biomedical Imaging. London, GB: IEEE Computer Society.
MLA:
Fischer, David Julian, Markus Eckstein, and Andreas Kist. "Latent Space Modelling of a Biomedical Multimodal Dataset." Proceedings of the 23rd IEEE International Symposium on Biomedical Imaging, ISBI 2026, London IEEE Computer Society, 2026.
BibTeX: Download