Lite2Relight: 3D-aware Single Image Portrait Relighting
Rao P, Fox G, Meka A, Mallikarjun BR, Zhan F, Weyrich T, Bickel B, Pfister H, Matusik W, Elgharib M, Theobalt C, Weyrich T (2024)
Publication Type: Conference contribution
Publication year: 2024
Publisher: Association for Computing Machinery, Inc
Conference Proceedings Title: Proceedings - SIGGRAPH 2024 Conference Papers
Event location: Denver, CO
ISBN: 9798400705250
Abstract
Achieving photorealistic 3D view synthesis and relighting of human portraits is pivotal for advancing AR/VR applications. Existing methodologies in portrait relighting demonstrate substantial limitations in terms of generalization and 3D consistency, coupled with inaccuracies in physically realistic lighting and identity preservation. Furthermore, personalization from a single view is difficult to achieve and often requires multiview images during the testing phase or involves slow optimization processes. This paper introduces Lite2Relight, a novel technique that can predict 3D consistent head poses of portraits while performing physically plausible light editing at interactive speed. Our method uniquely extends the generative capabilities and efficient volumetric representation of EG3D, leveraging a lightstage dataset to implicitly disentangle face reflectance and perform relighting under target HDRI environment maps. By utilizing a pre-trained geometry-aware encoder and a feature alignment module, we map input images into a relightable 3D space, enhancing them with a strong face geometry and reflectance prior. Through extensive quantitative and qualitative evaluations, we show that our method outperforms the state-of-the-art methods in terms of efficacy, photorealism, and practical application. This includes producing 3D-consistent results of the full head, including hair, eyes, and expressions. Lite2Relight paves the way for large-scale adoption of photorealistic portrait editing in various domains, offering a robust, interactive solution to a previously constrained problem.
Authors with CRIS profile
Involved external institutions
Max Planck Institute for Informatics / Max-Planck-Institut für Informatik
Germany (DE)
Harvard University
United States (USA) (US)
Massachusetts Institute of Technology (MIT)
United States (USA) (US)
Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich
Switzerland (CH)
Google Ireland Limited
Ireland (IE)
Germany (DE)
Harvard University
United States (USA) (US)
Massachusetts Institute of Technology (MIT)
United States (USA) (US)
Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich
Switzerland (CH)
Google Ireland Limited
Ireland (IE)
How to cite
APA:
Rao, P., Fox, G., Meka, A., Mallikarjun, B.R., Zhan, F., Weyrich, T.,... Weyrich, T. (2024). Lite2Relight: 3D-aware Single Image Portrait Relighting. In Stephen N. Spencer (Eds.), Proceedings - SIGGRAPH 2024 Conference Papers. Denver, CO, US: Association for Computing Machinery, Inc.
MLA:
Rao, Pramod, et al. "Lite2Relight: 3D-aware Single Image Portrait Relighting." Proceedings of the SIGGRAPH 2024 Conference Papers, Denver, CO Ed. Stephen N. Spencer, Association for Computing Machinery, Inc, 2024.
BibTeX: Download