Towards markerless surgical tool and hand pose estimation
Hein J, Seibold M, Bogo F, Farshad M, Pollefeys M, Furnstahl P, Navab N (2021)
Publication Type: Journal article
Publication year: 2021
Journal
Book Volume: 16
Pages Range: 799-808
Journal Issue: 5
DOI: 10.1007/s11548-021-02369-2
Abstract
Purpose: : Tracking of tools and surgical activity is becoming more and more important in the context of computer assisted surgery. In this work, we present a data generation framework, dataset and baseline methods to facilitate further research in the direction of markerless hand and instrument pose estimation in realistic surgical scenarios. Methods: : We developed a rendering pipeline to create inexpensive and realistic synthetic data for model pretraining. Subsequently, we propose a pipeline to capture and label real data with hand and object pose ground truth in an experimental setup to gather high-quality real data. We furthermore present three state-of-the-art RGB-based pose estimation baselines. Results: : We evaluate three baseline models on the proposed datasets. The best performing baseline achieves an average tool 3D vertex error of 16.7 mm on synthetic data as well as 13.8 mm on real data which is comparable to the state-of-the art in RGB-based hand/object pose estimation. Conclusion: : To the best of our knowledge, we propose the first synthetic and real data generation pipelines to generate hand and object pose labels for open surgery. We present three baseline models for RGB based object and object/hand pose estimation based on RGB frames. Our realistic synthetic data generation pipeline may contribute to overcome the data bottleneck in the surgical domain and can easily be transferred to other medical applications.
Involved external institutions
University of Zurich / Universität Zürich (UZH)
Switzerland (CH)
Microsoft Corporation
United States (USA) (US)
Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich
Switzerland (CH)
Technische Universität München (TUM)
Germany (DE)
Switzerland (CH)
Microsoft Corporation
United States (USA) (US)
Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich
Switzerland (CH)
Technische Universität München (TUM)
Germany (DE)
How to cite
APA:
Hein, J., Seibold, M., Bogo, F., Farshad, M., Pollefeys, M., Furnstahl, P., & Navab, N. (2021). Towards markerless surgical tool and hand pose estimation. International Journal of Computer Assisted Radiology and Surgery, 16(5), 799-808. https://doi.org/10.1007/s11548-021-02369-2
MLA:
Hein, Jonas, et al. "Towards markerless surgical tool and hand pose estimation." International Journal of Computer Assisted Radiology and Surgery 16.5 (2021): 799-808.
BibTeX: Download