Face2Face: Real-time face capture and reenactment of RGB videos

Thies J, Zollhofer M, Stamminger M, Theobalt C, Niessner M (2019)

Publication Type: Journal article

Publication year: 2019

Journal

Communications of the ACM ACM Association for Computing Machinery

Book Volume: 62

Pages Range: 96-104

Journal Issue: 1

DOI: 10.1145/3292039

Abstract

Face2Face is an approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time. This live setup has also been shown at SIGGRAPH Emerging Technologies 2016, by Thies et al. 20 where it won the Best in Show Award.

Authors with CRIS profile

Marc Stamminger Chair of Visual Computing

Involved external institutions

Stanford University US United States (USA) (US) Technische Universität München (TUM) DE Germany (DE) Max Planck Institute for Informatics / Max-Planck-Institut für Informatik DE Germany (DE)

How to cite

APA:

Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2019). Face2Face: Real-time face capture and reenactment of RGB videos. Communications of the ACM, 62(1), 96-104. https://dx.doi.org/10.1145/3292039

MLA:

Thies, Justus, et al. "Face2Face: Real-time face capture and reenactment of RGB videos." Communications of the ACM 62.1 (2019): 96-104.

BibTeX: Download