UniTorch - Integrating Neural Rendering into Unity

Fink L, Keitel D, Stamminger M, Keinert J (2023)

Publication Language: English

Publication Type: Conference contribution

Publication year: 2023

Journal

Lecture Notes in Computer Science Springer Verlag

Publisher: Springer

Series: Lecture Notes in Computer Science

City/Town: Cham

Book Volume: 14361

Pages Range: 315-328

Conference Proceedings Title: Advances in Visual Computing. 18th International Symposium, ISVC 2023, Lake Tahoe, NV, USA, October 16–18, 2023, Proceedings, Part I

Event location: Lake Tahoe, NV US

ISBN: 9783031479687

DOI: 10.1007/978-3-031-47969-4_25

Abstract

Neural rendering techniques have gained significant attention in recent years for their ability to generate highly realistic and immersive visual content. This paper discusses the current state of game engines regarding their ability to integrate neural modules within their pipelines. We exemplarily chose the popular game engine Unity and the deep learning library LibTorch. As we found a severe gap between commonly used auto-diff, deployment and rendering frameworks regarding interoperability and performance, we designed UniTorch, a plug-in that allows native access from Unity to Torch. We explore the practical integration of neural rendering methods by faithfully reimplementing and extending state-of-the-art methods. We provide detailed implementation guidelines and use it as means to reveal the mentioned gaps through extensive benchmarking experiments.

Authors with CRIS profile

Laura Fink Chair of Visual Computing (LGDV) Marc Stamminger Chair of Visual Computing (LGDV)

Involved external institutions

Fraunhofer-Institut für Integrierte Schaltungen (IIS) DE Germany (DE)

How to cite

APA:

Fink, L., Keitel, D., Stamminger, M., & Keinert, J. (2023). UniTorch - Integrating Neural Rendering into Unity. In George Bebis, Golnaz Ghiasi, Yi Fang, Andrei Sharf, Yue Dong, Chris Weaver, Zhicheng Leo, Joseph J. LaViola Jr., Luv Kohli (Eds.), Advances in Visual Computing. 18th International Symposium, ISVC 2023, Lake Tahoe, NV, USA, October 16–18, 2023, Proceedings, Part I (pp. 315-328). Lake Tahoe, NV, US: Cham: Springer.

MLA:

Fink, Laura, et al. "UniTorch - Integrating Neural Rendering into Unity." Proceedings of the 18th International Symposium on Visual Computing, ISVC 2023, Lake Tahoe, NV Ed. George Bebis, Golnaz Ghiasi, Yi Fang, Andrei Sharf, Yue Dong, Chris Weaver, Zhicheng Leo, Joseph J. LaViola Jr., Luv Kohli, Cham: Springer, 2023. 315-328.

BibTeX: Download