Efficient Prompt-Guided Image Segmentation with Frozen Vision-Language Models

Sarker K, Islam MZ, Antor JH, Sorker MS, Kabir H (2026)

Publication Type: Conference contribution

Publication year: 2026

Publisher: IEEE Computer Society

Conference Proceedings Title: International Conference of Young Specialists on Micro/Nanotechnologies and Electron Devices, EDM

Event location: Novosibirsk RU

ISBN: 9798331547950

DOI: 10.1109/EDM69524.2026.11632232

Abstract

Prompt-guided image segmentation enables flexible visual understanding through natural language and has potential utility as an instructional scaffold in human-AI collaborative environments. Existing methods often require fine-tuning large backbones, increasing computational costs and limiting accessibility in resource-constrained settings. This paper investigates whether frozen vision-language models can support both category-specific and open-vocabulary segmentation tasks using a lightweight convolutional prediction head. An efficient architecture is proposed that combines a frozen CLIP backbone (88.87M total parameters) with a compact 3.1M-parameter convolutional head. Rather than claiming algorithmic novelty, the contribution is an engineering study of the accuracy-efficiency trade-off. Experiments on Oxford-IIIT Pet, ECSSD, and DUTS-TE are conducted under a unified evaluation pipeline with three random seeds (42/43/44). The proposed model achieves 72.92 ± 1.03%, 66.55 ± 1.62%, and 62.86 ± 0.16% mIoU on the three datasets, outperforming all re-run baselines (UNet, FastSAM, MobileSAM) in the same pipeline. A direct VRAM/FPS/mIoU comparison with those baselines is provided. Pedagogical impact is treated as a deployment hypothesis requiring future user studies rather than as a demonstrated outcome.

Involved external institutions

Inha University / 인하대학교(仁荷大學校 KR Korea, Republic of (KR) Technische Universität Chemnitz DE Germany (DE) Fachhochschule Schmalkalden DE Germany (DE)

How to cite

APA:

Sarker, K., Islam, M.Z., Antor, J.H., Sorker, M.S., & Kabir, H. (2026). Efficient Prompt-Guided Image Segmentation with Frozen Vision-Language Models. In International Conference of Young Specialists on Micro/Nanotechnologies and Electron Devices, EDM. Novosibirsk, RU: IEEE Computer Society.

MLA:

Sarker, Karno, et al. "Efficient Prompt-Guided Image Segmentation with Frozen Vision-Language Models." Proceedings of the 27th International Conference of Young Professionals in Electron Devices and Materials, EDM 2026, Novosibirsk IEEE Computer Society, 2026.

BibTeX: Download