Efficient Rendering for Light Field Displays using Tailored Projective Mappings - Fink, Laura; Strobel, Svenja; Franke, Linus; Stamminger, Marc (Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2023)
A standard paradigm when rendering for parallax-based light field displays is to render multiple, slightly offset views first and to interweave these afterwards. In practice, more than 40 views of preferably high resolution need to be rendered per frame to achieve acceptable visual quality. The total amount of rendered pixels may consequently exceed the native resolution of the display by far. Increased memory consumption and sub-optimal render times are direct consequences. In this paper, we examine where pixels are "wasted"and present novel projective mappings for the virtual camera system that are custom tailored to such displays. Thus, we alleviate the aforementioned issues and show significant performance improvements regarding render time and memory consumption, while having only minor impact on visual quality. As we mainly touch the projective mapping of the virtual camera, our method is lean and can easily be integrated in existing rendering pipelines with minimal side effects.
CRIS-Team Scopus Importer:2023-06-02
Keywords: Light Field Display; Projection Matrix; Rendering; Super Sampling

Hybrid Mono-Stereo Rendering in Virtual Reality - Fink, Laura; Hensel, Nora; Markov-Vetter, Daniela; Weber, Christoph; Staadt, Oliver; Stamminger, Marc (IEEE Virtual Reality, 2019)

Inovis: Instant Novel-View Synthesis - Harrer, Mathias; Franke, Linus; Fink, Laura; Stamminger, Marc; Weyrich, Tim (ACM SIGGRAPH Asia 2023 Conference Proceedings, 2023)
Novel-view synthesis is an ill-posed problem in that it requires inference of previously unseen information. Recently, reviving the traditional field of image-based rendering, neural methods proved particularly suitable for this interpolation/extrapolation task; however, they often require a-priori scene-completeness or costly preprocessing steps and generally suffer from long (scene-specific) training times. Our work draws from recent progress in neural spatio-temporal supersampling to enhance a state-of-the-art neural renderer's ability to infer novel-view information at inference time. We adapt a supersampling architecture [Xiao et al. 2020], which resamples previously rendered frames, to instead recombine nearby camera images in a multi-view dataset. These input frames are warped into a joint target frame, guided by the most recent (point-based) scene representation, followed by neural interpolation. The resulting architecture gains sufficient robustness to significantly improve transferability to previously unseen datasets. In particular, this enables novel applications for neural rendering where dynamically streamed content is directly incorporated in a (neural) image-based reconstruction of a scene. As we will show, our method reaches state-of-the-art performance when compared to previous works that rely on static and sufficiently densely sampled scenes; in addition, we demonstrate our system's particular suitability for dynamically streamed content, where our approach is able to produce high-fidelity novel-view synthesis even with significantly fewer available frames than competing neural method
Keywords: Novel-View Synthesis; Point-based Graphics; Neural Rendering; Image-based Rendering; Reconstruction

Light field processing for media applications - Goldmann, Florian; Keinert, Joachim; Fink, Laura; Gul, Muhammad Shahzeb Khan; Jaschke, Tobias; Ziegler, Matthias; Baetz, Michel; Fößel, Siegfried; Prappacher, Nico (Immersive Video Technologies, 2023)

LiveNVS: Neural View Synthesis on Live RGB-D Streams - Fink, Laura; Rückert, Darius; Franke, Linus; Keinert, Joachim; Stamminger, Marc (Siggraph Asia 2023 Proceedings, 2023)
Existing real-time RGB-D reconstruction approaches, like Kinect Fusion, lack real-time photo-realistic visualization. This is due to noisy, oversmoothed or incomplete geometry and blurry textures which are fused from imperfect depth maps and camera poses. Recent neural rendering methods can overcome many of such artifacts but are mostly optimized for offline usage, hindering the integration into a live reconstruction pipeline. In this paper, we present LiveNVS, a system that allows for neural novel view synthesis on a live RGB-D input stream with very low latency and real-time rendering. Based on the RGB-D input stream, novel views are rendered by projecting neural features into the target view via a densely fused depth map and aggregating the features in image-space to a target feature map. A generalizable neural network then translates the target feature map into a high-quality RGB image. LiveNVS achieves state-of-the-art neural rendering quality of unknown scenes during capturing, allowing users to virtually explore the scene and assess reconstruction quality in real-time.

LumiPath – Towards Real-Time Physically-Based Rendering on Embedded Devices - Fink, Laura; Lee, Sing Chun; Wu, Jie Ying; Liu, Xingtong; Song, Tianyu; Velikova, Yordanka; Stamminger, Marc; Navab, Nassir; Unberath, Mathias (Lecture Notes in Computer Science, 2019)

Time‐Warped Foveated Rendering for Virtual Reality Headsets - Franke, Linus; Fink, Laura; Martschinke, Jana; Selgrad, Kai; Stamminger, Marc (Computer Graphics Forum, 2020)
Rendering in real time for virtual reality headsets with high user immersion is challenging due to strict framerate constraints as well as due to a low tolerance for artefacts. Eye tracking‐based foveated rendering presents an opportunity to strongly increase performance without loss of perceived visual quality. To this end, we propose a novel foveated rendering method for virtual reality headsets with integrated eye tracking hardware. Our method comprises recycling pixels in the periphery by spatio‐temporally reprojecting them from previous frames. Artefacts and disocclusions caused by this reprojection are detected and re‐evaluated according to a confidence value that is determined by a newly introduced formalized perception‐based metric, referred to as confidence function. The foveal region, as well as areas with low confidence values, are redrawn efficiently, as the confidence value allows for the delicate regulation of hierarchical geometry and pixel culling. Hence, the average primitive processing and shading costs are lowered dramatically. Evaluated against regular rendering as well as established foveated rendering methods, our approach shows increased performance in both cases. Furthermore, our method is not restricted to static scenes and provides an acceleration structure for post‐processing passes.<br
Keywords: perceptually based rendering; real‐time rendering; rendering immersive VR; virtual environments

TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering - Franke, Linus; Rückert, Darius; Fink, Laura; Stamminger, Marc (Computer Graphics Forum, 2024)
Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [KKLD23] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [RFS22] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage. The project page is located at: https://lfranke.github.io/trips.
CRIS-Team Scopus Importer:2024-05-10
Keywords: CCS Concepts; Image-based rendering; Reconstruction; • Computing methodologies → Rendering

UniTorch - Integrating Neural Rendering into Unity - Fink, Laura; Keitel, Daniel; Stamminger, Marc; Keinert, Joachim (Advances in Visual Computing. 18th International Symposium, ISVC 2023, Lake Tahoe, NV, USA, October 16–18, 2023, Proceedings, Part I, 2023)
Neural rendering techniques have gained significant attention in recent years for their ability to generate highly realistic and immersive visual content. This paper discusses the current state of game engines regarding their ability to integrate neural modules within their pipelines. We exemplarily chose the popular game engine Unity and the deep learning library LibTorch. As we found a severe gap between commonly used auto-diff, deployment and rendering frameworks regarding interoperability and performance, we designed UniTorch, a plug-in that allows native access from Unity to Torch. We explore the practical integration of neural rendering methods by faithfully reimplementing and extending state-of-the-art methods. We provide detailed implementation guidelines and use it as means to reveal the mentioned gaps through extensive benchmarking experiments.
CRIS-Team Scopus Importer:2024-12-16

VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering - Franke, Linus; Rückert, Darius; Fink, Laura; Innmann, Matthias; Stamminger, Marc (ACM SIGGRAPH Asia 2023 Conference Proceedings, 2023)
In the last few years, deep neural networks opened the doors for big advances in novel view synthesis. Many of these approaches are based on a (coarse) proxy geometry obtained by structure from motion algorithms. Small deficiencies in this proxy can be fixed by neural rendering, but larger holes or missing parts, as they commonly appear for thin structures or for glossy regions, still lead to distracting artifacts and temporal instability. In this paper, we present a novel neural-rendering-based approach to detect and fix such deficiencies. As a proxy, we use a point cloud, which allows us to easily remove outlier geometry and to fill in missing geometry without complicated topological operations. Keys to our approach are (i) a differentiable, blending point-based renderer that can blend out redundant points, as well as (ii) the concept of Visual Error Tomography (VET), which allows us to lift 2D error maps to identify 3D-regions lacking geometry and to spawn novel points accordingly. Furthermore, (iii) by adding points as nested environment maps, our approach allows us to generate high-quality renderings of the surroundings in the same pipeline. In our results, we show that our approach can improve the quality of a point cloud obtained by structure from motion and thus increase novel view synthesis quality significantly. In contrast to point growing techniques, the approach can also fix large-scale holes and missing thin structures effectively. Rendering quality outperforms state-of-the-art methods and temporal stability is significantly improved, while rendering is possible at real-time frame rates. <

VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points - Franke, Linus; Fink, Laura; Stamminger, Marc (Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2025)
Recent advances in novel view synthesis have demonstrated impressive results in fast photorealistic scene rendering through differentiable point rendering, either via Gaussian Splatting (3DGS) [Kerbl and Kopanas et al. 2023] or neural point rendering [Aliev et al. 2020]. Unfortunately, these directions require either a large number of small Gaussians or expensive per-pixel post-processing for reconstructing fine details, which negatively impacts rendering performance. To meet the high performance demands of virtual reality (VR) systems, primitive or pixel counts therefore must be kept low, affecting visual quality. In this paper, we propose a novel hybrid approach based on foveated rendering as a promising solution that combines the strengths of both point rendering directions regarding performance sweet spots. Analyzing the compatibility with the human visual system, we find that using a low-detailed, few primitive smooth Gaussian representation for the periphery is cheap to compute and meets the perceptual demands of peripheral vision. For the fovea only, we use neural points with a convolutional neural network for the small pixel footprint, which provides sharp, detailed output within the rendering budget. This combination also allows for synergistic method accelerations with point occlusion culling and reducing the demands on the neural network. Our evaluation confirms that our approach increases sharpness and details compared to a standard VR-ready 3DGS configuration, and participants of a user study overwhelmingly preferred our method. Our system meets the necessary performance requirements for real-time VR interactions, ultimately enhancing the user's immersive experience. The project page can be found at: https://lfranke.github.io/vr_splatti