Progidiff: Prompt-Guided Diffusion-Based Medical Image Segmentation

Lin Y, Xu M, Hölle M, Prabhakar C, Maier A, Belagiannis V, Menze B, Shit S (2026)

Publication Type: Conference contribution

Publication year: 2026

Journal

Proceedings - International Symposium on Biomedical Imaging

Publisher: IEEE Computer Society

Book Volume: 2026-April

Conference Proceedings Title: Proceedings - International Symposium on Biomedical Imaging

Event location: London GB

ISBN: 9798331577636

DOI: 10.1109/ISBI61048.2026.11515715

Abstract

Widely adopted medical image segmentation methods, although efficient, are primarily deterministic and remain poorly amenable to natural language prompts. Thus, they lack the capability to estimate multiple proposals, human interaction, and cross-modality adaption. Recently, text-toimage diffusion models have shown potential to bridge the gap. However, training them from scratch requires a large dataset-a limitation for medical image segmentation. Furthermore, they are often limited to binary segmentation and cannot be conditioned on a natural language prompt. To this end, we propose a novel framework called ProGiDiff that leverages existing image generation models for medical image segmentation purposes. Specifically, we propose a ControlNet-style conditioning mechanism with a custom encoder, suitable for image conditioning, to steer a pre-trained diffusion model to output segmentation masks. It naturally extends to a multi-class setting simply by prompting the target organ. Our experiment on organ segmentation from CT images demonstrates strong performance compared to previous methods and could greatly benefit from an expert-in-theloop setting to leverage multiple proposals. Importantly, we demonstrate that the learned conditioning mechanism can be easily transferred through low-rank, few-shot adaptation to segment MR images.

Authors with CRIS profile

Marc Hölle Professur für Maschinelles Lernen in der Signalverarbeitung Andreas Maier Lehrstuhl für Informatik 5 (Mustererkennung) Vasileios Belagiannis Professur für Maschinelles Lernen in der Signalverarbeitung

Involved external institutions

University of Zurich / Universität Zürich (UZH) CH Switzerland (CH)

How to cite

APA:

Lin, Y., Xu, M., Hölle, M., Prabhakar, C., Maier, A., Belagiannis, V.,... Shit, S. (2026). Progidiff: Prompt-Guided Diffusion-Based Medical Image Segmentation. In Proceedings - International Symposium on Biomedical Imaging. London, GB: IEEE Computer Society.

MLA:

Lin, Yuan, et al. "Progidiff: Prompt-Guided Diffusion-Based Medical Image Segmentation." Proceedings of the 23rd IEEE International Symposium on Biomedical Imaging, ISBI 2026, London IEEE Computer Society, 2026.

BibTeX: Download