Generation of Controllable and Photorealistic Synthetic Cataract Surgery Images: Blending 3D Models and Real-World Data
Peter R, Oberschulte E, Wu E, Vaidya A, Lindemeier T, Tagliabue E, Mathis-Ullrich F (2026)
Publication Type: Conference contribution
Publication year: 2026
Journal
Publisher: Springer Science and Business Media Deutschland GmbH
Book Volume: 16085 LNCS
Pages Range: 97-106
Conference Proceedings Title: Lecture Notes in Computer Science
Event location: Daejeon, KOR
ISBN: 9783032055729
DOI: 10.1007/978-3-032-05573-6_10
Abstract
When generating data-driven algorithms for surgical scene understanding, there is a trade-off between utilizing real-world image data, which is often costly to acquire and to annotate and lacks certain critical information like depth maps, and employing model-based synthetic images, which typically introduce a domain gap. This gap is often particularly pronounced in visual appearance, making it challenging for models trained on synthetic data to transfer effectively to real-world scenarios. Our approach seeks to bridge this gap by combining parameterizable 3D models of anatomical structures with real-world textural data, thereby minimizing the visual domain gap. We propose a method for generating synthetic cataract surgery images from surgical stereo microscopes that integrates 3D models with real-world textures for anatomical structures. Our approach accommodates variations in eye geometry, surface texture, surgical instrument shapes and placements, camera parameters and positions, as well as lighting conditions. Additionally, we detail our approach for generating annotations, including depth maps and semantic segmentation masks, specifically designed to address cornea-induced distortions present in images of the human eye. Our synthetic images accurately reflect the optical characteristics of the human eye, providing reference data for depth estimation and 3D reconstruction tasks unattainable from real-world sources. The pipeline supports the creation of large-scale datasets with high variance, facilitating the development of robust deep learning models for scene understanding in computer- and robot-assisted cataract surgery.
Authors with CRIS profile
Involved external institutions
Germany (DE)How to cite
APA:
Peter, R., Oberschulte, E., Wu, E., Vaidya, A., Lindemeier, T., Tagliabue, E., & Mathis-Ullrich, F. (2026). Generation of Controllable and Photorealistic Synthetic Cataract Surgery Images: Blending 3D Models and Real-World Data. In Virginia Fernandez, Lianrui Zuo, Samuel W. Remedios, David Wiesner, Adrià Casamitjana (Eds.), Lecture Notes in Computer Science (pp. 97-106). Daejeon, KOR: Springer Science and Business Media Deutschland GmbH.
MLA:
Peter, Rebekka, et al. "Generation of Controllable and Photorealistic Synthetic Cataract Surgery Images: Blending 3D Models and Real-World Data." Proceedings of the 10th International Workshop on Simulation and Synthesis in Medical Imaging, SASHIMI 2025, held in conjunction with the 28th International conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025, Daejeon, KOR Ed. Virginia Fernandez, Lianrui Zuo, Samuel W. Remedios, David Wiesner, Adrià Casamitjana, Springer Science and Business Media Deutschland GmbH, 2026. 97-106.
BibTeX: Download