Real-time Non-rigid Reconstruction using an RGB-D Camera

Izadi S, Rhemann C, Zach C, Fisher M, Wu C, Fitzgibbon A, Loop C, Theobald C, Stamminger M, Nießner M, Zollhöfer M (2014)

Publication Type: Journal article

Publication year: 2014

Journal

Acm Transactions on Graphics Association for Computing Machinery (ACM)

Book Volume: 33

Pages Range: 156:1-12

Journal Issue: 4

DOI: 10.1145/2601097.2601165

Abstract

We present a combined hardware and software solution for marker-less reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time. Our system uses a single self-contained stereo camera unit built from off-the-shelf components and consumer graphics hardware to generate spatio-temporally coherent 3D models at 30 Hz. A new stereo matching algorithm estimates real-time RGB-D data. We start by scanning a smooth template model of the subject as they move rigidly. This geometric surface prior avoids strong scene assumptions, such as a kinematic human skeleton or a parametric shape model. Next, a novel GPU pipeline performs non-rigid registration of live RGB-D data to the smooth template using an extended non-linear as-rigid-as-possible (ARAP) framework. High-frequency details are fused onto the final mesh using a linear deformation model. The system is an order of magnitude faster than state-of-the-art methods, while matching the quality and robustness of many offline algorithms. We show precise real-time reconstructions of diverse scenes, including: large deformations of users' heads, hands, and upper bodies; fine-scale wrinkles and folds of skin and clothing; and non-rigid interactions performed by users on flexible objects such as toys. We demonstrate how acquired models can be used for many interactive scenarios, including re-texturing, online performance capture and preview, and real-time shape and motion re-targeting. Copyright © ACM.

Authors with CRIS profile

Marc Stamminger Chair of Visual Computing Matthias Nießner Chair of Visual Computing Michael Zollhöfer Chair of Visual Computing

Involved external institutions

Microsoft Corporation US United States (USA) (US) Stanford University US United States (USA) (US)

How to cite

APA:

Izadi, S., Rhemann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A.,... Zollhöfer, M. (2014). Real-time Non-rigid Reconstruction using an RGB-D Camera. Acm Transactions on Graphics, 33(4), 156:1-12. https://dx.doi.org/10.1145/2601097.2601165

MLA:

Izadi, Sharam, et al. "Real-time Non-rigid Reconstruction using an RGB-D Camera." Acm Transactions on Graphics 33.4 (2014): 156:1-12.

BibTeX: Download