Automatic Transfer of Landmarks on Human Skulls using GPU-based Non-rigid Registration - Innmann, Matthias; Erhardt, Philipp; Schütz, Daniel; Greiner, Günther (GCH 2017 - Eurographics Workshop on Graphics and Cultural Heritage, 2017)
In this work, we present a novel approach to automatically transfer landmarks from a template mesh of a human skull to other meshes obtained via 3D scanning. Since previous methods rely on user input or only work on a subset of the data, these algorithms are not suited for large databases. Our system is designed to work for arbitrary meshes of human skulls, i.e. having artifacts like incomplete geometry or being non-watertight. Since the input data has no common orientation, we first apply a rigid coarse registration followed by a refinement. Afterwards, the remaining geometric deviation is removed by non-rigidly deforming one mesh into the other. With this precise geometric mapping, arbitrary landmarks can be transferred easily. To ensure efficient computation, we use a highly optimized GPU implementation to solve arising optimization problems. We apply our method to a dataset consisting of 1200 models acquired via structured light scanning and evaluate its accuracy on a subset of these models.

BRDF-Reconstruction in Photogrammetry Studio Setups - Innmann, Matthias; Süßmuth, Jochen; Stamminger, Marc (IEEE Winter Conference on Applications of Computer Vision (WACV) 2020, 2020)

FragmentFusion: A Light-weight SLAM Pipeline for Dense Reconstruction - Rückert, Darius; Innmann, Matthias; Stamminger, Marc (ISMAR 2019, 2019)
We present FragmentFusion, a real-time dense reconstruction pipeline that combines sparse camera tracking with image-space volumetric fusion. The tracking is based on ORB-SLAM, which constructs and optimizes a sparse global map of 3D points and keyframes. We transform each of these keyframes into decimated meshes and render them from the estimated viewpoint. Fusion is performed in the pixel shader by exploiting atomic operations on a packed data structure. This eliminates the need of a 3D voxel grid making FragmentFusion very flexible at large scenes and varying scales. Moreover, since all keyframes are fused on-the-fly, we can use bundle adjustment and loop-closure without expensive volu- metric re-integration. FragmentFusion is lightweight in terms of compute power and memory consumption. It can easily fuse sev- eral hundreds of keyframes in real time, in a quality comparable to other approaches. We achieve real-time frame rates on a note- book at around 20% CPU and GPU utilization and low memory consumption. <br

NRMVS: Non-Rigid Multi-View Stereo - Innmann, Matthias; Kim, Kihwan; Gu, Jinwei; Niessner, Matthias; Loop, Charles; Stamminger, Marc; Kautz, Jan (arXiv, 2019)
Scene reconstruction from unorganized RGB images is an important task in many computer vision applications. Multi-view Stereo (MVS) is a common solution in photogrammetry applications for the dense reconstruction of a static scene. The static scene assumption, however, limits the general applicability of MVS algorithms, as many day-to-day scenes undergo non-rigid motion, e.g., clothes, faces, or human bodies. In this paper, we open up a new challenging direction: dense 3D reconstruction of scenes with non-rigid changes observed from arbitrary, sparse, and wide-baseline views. We formulate the problem as a joint optimization of deformation and depth estimation, using deformation graphs as the underlying representation. We propose a new sparse 3D to 2D matching technique, together with a dense patch-match evaluation scheme to estimate deformation and depth with photometric consistency. We show that creating a dense 4D structure from a few RGB images with non-rigid changes is possible, and demonstrate that our method can be used to interpolate novel deformed scenes from various combinations of these deformation estimates derived from the sparse views.

NRMVS: Non-Rigid Multi-View Stereo - Innmann, Matthias; Kim, Kihwan; Gu, Jinwei; Nießner, Matthias; Loop, Charles; Stamminger, Marc; Kautz, Jan (IEEE Winter Conference on Applications of Computer Vision (WACV), 2020)
Multi-view Stereo (MVS) is a common solution in photogrammetry applications for the dense reconstruction of a static scene from images. The static scene assumption, however, limits the general applicability of MVS algorithms, as many day-to-day scenes undergo non-rigid motion, e.g., clothes, faces, or human bodies. In this paper, we open up a new challenging direction: Dense 3D reconstruction of scenes with non-rigid changes observed from a small number of images sparsely captured from different views with a single monocular camera, which we call non-rigid multi-view stereo (NRMVS) problem. We formulate this problem as a joint optimization of deformation and depth estimation, using deformation graphs as the underlying representation. We propose a new sparse 3D to 2D matching technique with a dense patch-match evaluation scheme to estimate the most plausible deformation field satisfying depth and photometric consistency. We show that a dense reconstruction of a scene with non-rigid changes from a few images is possible, and demonstrate that our method can be used to interpolate novel deformed scenes from various combinations of deformation estimates derived from the sparse view

Shading-based refinement on volumetric signed distance functions - Zollhöfer, Michael; Dai, Angela; Innmann, Matthias; Wu, Chenglei; Theobald, Christian; Nießner, Matthias; Stamminger, Marc (ACM Transactions on Graphics, 2015)
We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices. As the depth data of these sensors is noisy, truncated signed distance fields are typically used to regularize out the noise, which unfortunately leads to over-smoothed results. In our approach, we leverage RGB data to refine these reconstructions through shading cues, as color input is typically of much higher resolution than the depth data. As a result, we obtain reconstructions with high geometric detail, far beyond the depth resolution of the camera itself. Our core contribution is shading-based refinement directly on the implicit surface representation, which is generated from globally-aligned RGB-D images. We formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance. In order to enable the efficient reconstruction of sub-millimeter detail, we store and process our surface using a sparse voxel hashing scheme which we augment by introducing a grid hierarchy. A tailored GPU-based Gauss-Newton solver enables us to refine large shape models to previously unseen resolution within only a few second
Keywords: 3D reconstruction; Shading-based refinement

Spherical Fibonacci mapping - Keinert, Benjamin; Innmann, Matthias; Sänger, Michael; Stamminger, Marc (ACM Transactions on Graphics, 2015)
Spherical Fibonacci point sets yield nearly uniform point distributions on the unit sphere S ⊂ ℝ. The forward generation of these point sets has been widely researched and is easy to implement, such that they have been used in various applications. Unfortunately, the lack of an efficient mapping from points on the unit sphere to their closest spherical Fibonacci point set neighbors rendered them impractical for a wide range of applications, especially in computer graphics. Therefore, we introduce an inverse mapping from points on the unit sphere which yields the nearest neighbor in an arbitrarily sized spherical Fibonacci point set in constant time, without requiring any precomputations or table lookups. We show how to implement this inverse mapping on GPUs while addressing arising floating point precision problems. Further, we demonstrate the use of this mapping and its variants, and show how to apply it to fast unit vector quantization. Finally, we illustrate the means by which to modify this inverse mapping for texture mapping with smooth filter kernels and showcase its use in the field of procedural modelin
Keywords: Constant time; Inverse mapping; Spherical Fibonacci

VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering - Franke, Linus; Rückert, Darius; Fink, Laura; Innmann, Matthias; Stamminger, Marc (ACM SIGGRAPH Asia 2023 Conference Proceedings, 2023)
In the last few years, deep neural networks opened the doors for big advances in novel view synthesis. Many of these approaches are based on a (coarse) proxy geometry obtained by structure from motion algorithms. Small deficiencies in this proxy can be fixed by neural rendering, but larger holes or missing parts, as they commonly appear for thin structures or for glossy regions, still lead to distracting artifacts and temporal instability. In this paper, we present a novel neural-rendering-based approach to detect and fix such deficiencies. As a proxy, we use a point cloud, which allows us to easily remove outlier geometry and to fill in missing geometry without complicated topological operations. Keys to our approach are (i) a differentiable, blending point-based renderer that can blend out redundant points, as well as (ii) the concept of Visual Error Tomography (VET), which allows us to lift 2D error maps to identify 3D-regions lacking geometry and to spawn novel points accordingly. Furthermore, (iii) by adding points as nested environment maps, our approach allows us to generate high-quality renderings of the surroundings in the same pipeline. In our results, we show that our approach can improve the quality of a point cloud obtained by structure from motion and thus increase novel view synthesis quality significantly. In contrast to point growing techniques, the approach can also fix large-scale holes and missing thin structures effectively. Rendering quality outperforms state-of-the-art methods and temporal stability is significantly improved, while rendering is possible at real-time frame rates. <

VolumeDeform: Real-Time Volumetric Non-rigid Reconstruction - Innmann, Matthias; Zollhöfer, Michael; Nießner, Matthias; Theobald, Christian; Stamminger, Marc (ECCV 2016 - Proceedings of the European Conference on Computer Vision, 2016)
We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method builds up the scene model from scratch during the scanning process, thus it does not require a predefined shape template to start with. Geometry and motion are parameterized in a unified manner by a volumetric representation that encodes a distance field of the surface geometry as well as the non-rigid space deformation. Motion tracking is based on a set of extracted sparse color features in combination with a dense depth constraint. This enables accurate tracking and drastically reduces drift inherent to standard modelto- depth alignment. We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints. The problem is tackled in real-time at the camera’s capture rate using a data-parallel flip-flop optimization strategy. Our results demonstrate robust tracking even for fast motion and scenes that lack geometric feature