4D-OR: Semantic Scene Graphs for OR Domain Modeling
Ozsoy E, Ornek EP, Eck U, Czempiel T, Tombari F, Navab N (2022)
Publication Type: Conference contribution
Publication year: 2022
Journal
Publisher: Springer Science and Business Media Deutschland GmbH
Book Volume: 13437 LNCS
Pages Range: 475-485
Conference Proceedings Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Event location: Singapore, SGP
ISBN: 9783031164484
DOI: 10.1007/978-3-031-16449-1_45
Abstract
Surgical procedures are conducted in highly complex operating rooms (OR), comprising different actors, devices, and interactions. To date, only medically trained human experts are capable of understanding all the links and interactions in such a demanding environment. This paper aims to bring the community one step closer to automated, holistic and semantic understanding and modeling of OR domain. Towards this goal, for the first time, we propose using semantic scene graphs (SSG) to describe and summarize the surgical scene. The nodes of the scene graphs represent different actors and objects in the room, such as medical staff, patients, and medical equipment, whereas edges are the relationships between them. To validate the possibilities of the proposed representation, we create the first publicly available 4D surgical SSG dataset, 4D-OR, containing ten simulated total knee replacement surgeries recorded with six RGB-D sensors in a realistic OR simulation center. 4D-OR includes 6734 frames and is richly annotated with SSGs, human and object poses, and clinical roles. We propose an end-to-end neural network-based SSG generation pipeline, with a rate of success of 0.75 macro F1, indeed being able to infer semantic reasoning in the OR. We further demonstrate the representation power of our scene graphs by using it for the problem of clinical role prediction, where we achieve 0.85 macro F1. The code and dataset are publicly available at github.com/egeozsoy/4D-OR.
Involved external institutions
Germany (DE)How to cite
APA:
Ozsoy, E., Ornek, E.P., Eck, U., Czempiel, T., Tombari, F., & Navab, N. (2022). 4D-OR: Semantic Scene Graphs for OR Domain Modeling. In Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, Shuo Li (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 475-485). Singapore, SGP: Springer Science and Business Media Deutschland GmbH.
MLA:
Ozsoy, Ege, et al. "4D-OR: Semantic Scene Graphs for OR Domain Modeling." Proceedings of the 25th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2022, Singapore, SGP Ed. Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, Shuo Li, Springer Science and Business Media Deutschland GmbH, 2022. 475-485.
BibTeX: Download