Efficient Ray Tracing of Subdivision Surfaces using Tessellation Caching - Nießner, Matthias; Selgrad, Kai; Woop, Sven; Benthin, Carsten; et al. (Proceedings of the 7th High-Performance Graphics Conference, 2015)
UnivIS-Import:2015-07-08:Pub.2015.tech.IMMD.IMMD9.effici

Face2Face: Real-time Face Capture and Reenactment of RGB Videos - Thies, Justus; Zollhöfer, Michael; Stamminger, Marc; Theobald, Christian; Nießner, Matthias (Proc. Computer Vision and Pattern Recognition (CVPR), 2016)
We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real tim
UnivIS-Import:2016-06-01:Pub.2016.tech.IMMD.IMMD9.face2f
Keywords: GRK-1773

Face2Face: real-time face capture and reenactment of RGB videos - Thies, Justus; Zollhöfer, Michael; Stamminger, Marc; Theobalt, Christian; Nießner, Matthias (Communications of the ACM, 2019)

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality - Thies, Justus; Zollhöfer, Michael; Stamminger, Marc; Theobald, Christian; Nießner, Matthias (arXiv.org, 2016)
We introduce FaceVR, a novel method for gaze-aware facial reenactment in the Virtual Reality (VR) context. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD), as well as a new data-driven approach for eye tracking from monocular videos. In addition to these face reconstruction components, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances. For instance, we can alter facial expressions, change gaze directions, or remove the VR goggles in realistic re-renderings. In a live setup with a source and a target actor, we apply these newly-introduced algorithmic components. We assume that the source actor is wearing a VR device, and we capture his facial expressions and eye movement in real-time. For the target video, we mimic a similar tracking process; however, we use the source input to drive the animations of the target video, thus enabling gaze-aware facial reenactment. To render the modified target video on a stereo display, we augment our capture and reconstruction process with stereo data. In the end, FaceVR produces compelling results for a variety of applications, such as gaze-aware facial reenactment, reenactment in virtual reality, removal of VR goggles, and re-targeting of somebody's gaze direction in a video conferencing call.
UnivIS-Import:2017-01-09:Pub.2016.tech.IMMD.IMMD9.facevr

Image-guided Neural Object Rendering - Thies, Justus; Zollhöfer, Michael; Theobalt, Christian; Stamminger, Marc; Nießner, Matthias (Proceedings ICLR 2020, 2020)

Local Painting and Deformation of Meshes on the GPU - Stamminger, Marc; Schäfer, Henry; Keinert, Benjamin; Nießner, Matthias (Computer Graphics Forum, 2014)
UnivIS-Import:2015-03-09:Pub.2014.tech.IMMD.IMMD9.localp

Real-time 3D Reconstruction at Scale using Voxel Hashing - Izadi, Sharam; Zollhöfer, Michael; Nießner, Matthias; Stamminger, Marc (Acm Transactions on Graphics, 2013)
Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. The basic problem takes live overlapping depth maps as input and incrementally fuses these into a single 3D model. This is challenging particularly when real-time performance is desired without trading quality or scale. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure. Our system uses a simple spatial hashing scheme that compresses space, and allows for real-time access and updates of implicit surface data, without the need for a regular or hierarchical grid data structure. Surface data is only stored densely where measurements are observed. Additionally, data can be streamed efficiently in or out of the hash table, allowing for further scalability during sensor motion. We show interactive reconstructions of a variety of scenes, reconstructing both fine-grained details and large scale environments. We illustrate how all parts of our pipeline from depth map pre-processing, camera pose estimation, depth map fusion, and surface rendering are performed at real-time rates on commodity graphics hardware. We conclude with a comparison to current state-of-the-art online systems, illustrating improved performance and reconstruction qualit
UnivIS-Import:2015-03-09:Pub.2013.tech.IMMD.IMMD9.realti_5
Keywords: Data structure; GPU; Real-time reconstruction; Scalable

Real-Time Deformation of Subdivision Surfaces from Object Collisions - Buchenau, Christoph; Guthe, Michael; Nießner, Matthias; Stamminger, Marc; Schäfer, Henry; Keinert, Benjamin (Proceedings of the 6th High-Performance Graphics Conference, 2014)
We present a novel real-time approach for fine-scale surface deformations resulting from collisions. Deformations are represented by a high-resolution displacement function. When two objects collide, these offsets are updated directly on the GPU based on a dynamically generated binary voxelization of the overlap region. Consequently, we can handle collisions with arbitrary animated geometry. Our approach runs entirely on the GPU, avoiding costly CPU-GPU memory transfer and exploiting the GPU's computational power. Surfaces are rendered with the hardware tessellation unit, allowing for adaptively-rendered, high-frequency surface detail. Ultimately, our algorithm enables fine-scale surface deformations from geometry impact with very little computational overhead, running well below a millisecond even in complex scenes. As our results demonstrate, our approach is ideally suited to many real-time applications such as video games and authoring tool
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD9.realti_6
Keywords: Real-time Rendering; Subdivision Surfaces; Surface Deformation

Real-Time Deformation of Subdivision Surfaces on Object Collisions - Nießner, Matthias; Keinert, Benjamin; Schäfer, Henry; Stamminger, Marc (GPU Pro 6 Advanced Rendering Techniques, 2015)
UnivIS-Import:2016-02-10:Pub.2015.tech.IMMD.IMMD9.realti_3
Keywords: GRK-1773

Real-time Expression Transfer for Facial Reenactment - Thies, Justus; Zollhöfer, Michael; Nießner, Matthias; Valgaerts, Levi; Stamminger, Marc; Theobald, Christian (ACM Transactions on Graphics, 2015)
We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.
UnivIS-Import:2015-10-26:Pub.2015.tech.IMMD.IMMD9.realti_9
Keywords: GRK-1773

Real-time Non-rigid Reconstruction using an RGB-D Camera - Izadi, Sharam; Rhemann, Christoph; Zach, Christopher; Fisher, Matthew; Wu, Chenglei; Fitzgibbon, Andrew; Loop, Charles; Theobald, Christian; Stamminger, Marc; Nießner, Matthias; Zollhöfer, Michael (ACM Transactions on Graphics, 2014)
We present a combined hardware and software solution for marker-less reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time. Our system uses a single self-contained stereo camera unit built from off-the-shelf components and consumer graphics hardware to generate spatio-temporally coherent 3D models at 30 Hz. A new stereo matching algorithm estimates real-time RGB-D data. We start by scanning a smooth template model of the subject as they move rigidly. This geometric surface prior avoids strong scene assumptions, such as a kinematic human skeleton or a parametric shape model. Next, a novel GPU pipeline performs non-rigid registration of live RGB-D data to the smooth template using an extended non-linear as-rigid-as-possible (ARAP) framework. High-frequency details are fused onto the final mesh using a linear deformation model. The system is an order of magnitude faster than state-of-the-art methods, while matching the quality and robustness of many offline algorithms. We show precise real-time reconstructions of diverse scenes, including: large deformations of users' heads, hands, and upper bodies; fine-scale wrinkles and folds of skin and clothing; and non-rigid interactions performed by users on flexible objects such as toys. We demonstrate how acquired models can be used for many interactive scenarios, including re-texturing, online performance capture and preview, and real-time shape and motion re-targeting. Copyright © AC
UnivIS-Import:2015-03-09:Pub.2014.tech.IMMD.IMMD9.realti
Keywords: 3D scanning; Deformation; Depth camera; Non-rigid; Shape; Stereo matching; Surface reconstruction

Real-time Simulation of Human Vision using Temporal Compositing with CUDA on the GPU - Nießner, Matthias; Kuhnert, Nadine; Selgrad, Kai; Stamminger, Marc; Michelson, Georg (Proceedings 25th Workshop on Parallel Systems and Algorithms (PARS 2013), 2013)
We present a novel approach that simulates human vision including visual defects such as glaucoma by temporal composition of human vision in real-time on the GPU. Therefore, we determine eye focus points every time step and adapt the lens accommodation of our virtual eye model accordingly. The focal distance is then used to determine bluriness of observed scene regions; i.e., we compute defocus for all visible pixels. In order to simulate the visual memory we introduce a sharpness field where we integrate defocus values temporally. That allows for memorizing sharply perceived scene points. For visualization, we ray trace the virtual scene environment while incorporating depth of field based on the sharpness field data. Thus, our algorithm facilitates the simulation of human vision mimicing the visual memory. We consider this to be particularly useful for illustration purposes for patients with visual defects such as glaucoma. In order to run our algorithm in real-time we employ massively parallel graphics hardware.
UnivIS-Import:2015-04-16:Pub.2013.tech.IMMD.IMMD9.realti_9
Keywords: GRK-1773

State of the Art Report on Real-time Rendering with Hardware Tessellation - Loop, Charles; Schäfer, Henry; Stamminger, Marc; Keinert, Benjamin; Nießner, Matthias (State of the Art Reports, 2014)
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD9.stateo
Keywords: GRK-1773