Generalist foundation models from a multimodal dataset for 3D computed tomography

Hamamci IE, Er S, Wang C, Almas F, Simsek AG, Esirgun SN, Dogan I, Durugol OF, Hou B, Shit S, Dai W, Xu M, Reynaud H, Dasdelen MF, Wittmann B, Amiranashvili T, Simsar E, Simsar M, Erdemir EB, Alanbay A, Sekuboyina A, Lafci B, Kaplan A, Lu Z, Polacin M, Kainz B, Bluethgen C, Batmanghelich K, Ozdemir MK, Menze B (2026)

Publication Type: Journal article

Publication year: 2026

Journal

Nature Biomedical Engineering Nature Research

DOI: 10.1038/s41551-025-01599-y

Abstract

Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language–image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP’s vision encoder with a pretrained large language model, we create CT-CHAT, a vision–language foundational chat model for 3D chest CT volumes. Fine-tuned on over 2.7 million question–answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care.

Authors with CRIS profile

Hadrien Jean-Francois Reynaud
Bernhard Kainz

Involved external institutions

University of Zurich / Universität Zürich (UZH) CH Switzerland (CH) Istanbul Medipol University / İstanbul Medipol Üniversitesi TR Turkey (TR) US National Institutes of Health (NIH) US United States (USA) (US) Universitätsspital Zürich (USZ) CH Switzerland (CH) Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB) Boston University US United States (USA) (US) Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich CH Switzerland (CH)

How to cite

APA:

Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N.,... Menze, B. (2026). Generalist foundation models from a multimodal dataset for 3D computed tomography. Nature Biomedical Engineering. https://doi.org/10.1038/s41551-025-01599-y

MLA:

Hamamci, Ibrahim Ethem, et al. "Generalist foundation models from a multimodal dataset for 3D computed tomography." Nature Biomedical Engineering (2026).

BibTeX: Download