ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty

Wirth V, Liphardt AM, Coppers B, Bräunig J, Heinrich S, Leyendecker S, Kleyer A, Schett G, Vossiek M, Egger B, Stamminger M (2023)

Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2023

Pages Range: 2625-2633

Conference Proceedings Title: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops

Event location: Paris FR

Open Access Link: https://openaccess.thecvf.com/content/ICCV2023W/CVAMD/html/Wirth_ShaRPy_Shape_Reconstruction_and_Hand_Pose_Estimation_from_RGB-D_with_ICCVW_2023_paper.html

Abstract

Despite their potential, markerless hand tracking technologies are not yet applied in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. One reason is that the focus of most methods lies in the reconstruction of coarse, plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are required. Therefore, we propose ShaRPy, the first RGB-D Shape Reconstruction and hand Pose tracking system, which provides uncertainty estimates of the computed pose, e.g., when a finger is hidden or its estimate is inconsistent with the observations in the input, to guide clinical decision-making. Besides pose, ShaRPy approximates a personalized hand shape, promoting a more realistic and intuitive understanding of its digital twin. Our method requires only a light-weight setup with a single consumer-level RGB-D camera yet it is able to distinguish similar poses with only small joint angle deviations in a metrically accurate space. This is achieved by combining a data-driven dense correspondence predictor with traditional energy minimization. To bridge the gap between interactive visualization and biomedical simulation we leverage a parametric hand model in which we incorporate biomedical constraints and optimize for both, its pose and hand shape parameters. We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results of hand function assessments for activity monitoring of musculoskeletal diseases.

Authors with CRIS profile

Vanessa Wirth Chair of Visual Computing Anna-Maria Liphardt Department of Medicine 3 – Rheumatology and Immunology Birte Coppers Medizinische Fakultät Johanna Bräunig Lehrstuhl für Hochfrequenztechnik Simon Heinrich Institute of Applied Dynamics Sigrid Leyendecker Institute of Applied Dynamics Arnd Kleyer Department of Medicine 3 – Rheumatology and Immunology Georg Schett Lehrstuhl für Innere Medizin III Martin Vossiek Lehrstuhl für Hochfrequenztechnik Bernhard Egger Professur für Cognitive Computer Vision (Stiftungsprofessur) Marc Stamminger Chair of Visual Computing

Additional Organisation(s)

Sonderforschungsbereich 1483/1 Empathokinästhetische Sensorik – Sensortechniken und Datenanalyseverfahren zur empathokinästhetischen Modellbildung und Zustandsbestimmung FAU Profilzentrum Licht.Materie.Quantentechnologien (FAU LMQ)

Related research project(s)

Sensortechniken und Datenanalyseverfahren zur empathokinästhetischen Modellbildung und Zustandsbestimmung (SFB 1483 EmpkinS A01) Empatho-Kinaesthetic Sensor Technology (SFB 1483 EmpkinS) July 1, 2021 - June 30, 2025

How to cite

APA:

Wirth, V., Liphardt, A.-M., Coppers, B., Bräunig, J., Heinrich, S., Leyendecker, S.,... Stamminger, M. (2023). ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (pp. 2625-2633). Paris, FR.

MLA:

Wirth, Vanessa, et al. "ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty." Proceedings of the International Conference on Computer Vision (ICCV) Workshops, Paris 2023. 2625-2633.

BibTeX: Download