Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising
Hasselgren J, Hofmann N, Munkberg J (2022)
Publication Language: English
Publication Type: Conference contribution, Original article
Publication year: 2022
Pages Range: 14
Event location: New Orleans Convention Center (Hybrid)
URI: https://nvlabs.github.io/nvdiffrecmc/
Open Access Link: https://proceedings.neurips.cc/paper_files/paper/2022/hash/8fcb27984bf16ca03cad643244ec470d-Abstract-Conference.html
Abstract
Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that a more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials and lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline. This substantially improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become an integral part of high quality inverse rendering pipelines.
Authors with CRIS profile
Involved external institutions
How to cite
APA:
Hasselgren, J., Hofmann, N., & Munkberg, J. (2022). Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. In Proceedings of the Neural Information Processing Systems (NeurIPS 2022) (pp. 14). New Orleans Convention Center (Hybrid), US.
MLA:
Hasselgren, Jon, Nikolai Hofmann, and Jacob Munkberg. "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising." Proceedings of the Neural Information Processing Systems (NeurIPS 2022), New Orleans Convention Center (Hybrid) 2022. 14.
BibTeX: Download