Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

Schieber H, Duerr F, Schoen T, Beyerer J (2022)

Publication Type: Conference contribution

Publication year: 2022

Publisher: Institute of Electrical and Electronics Engineers Inc.

Book Volume: 2022-June

Pages Range: 375-381

Conference Proceedings Title: IEEE Intelligent Vehicles Symposium, Proceedings

Event location: Aachen, DEU

ISBN: 9781665488211

DOI: 10.1109/IV51971.2022.9827113

Abstract

Robust environment perception for autonomous vehicles is a tremendous challenge, which makes a diverse sensor set with e.g. camera, lidar and radar crucial. In the process of understanding the recorded sensor data, 3D semantic segmentation plays an important role. Therefore, this work presents a pyramid-based deep fusion architecture for lidar and camera to improve 3D semantic segmentation of traffic scenes. Individual sensor backbones extract feature maps of camera images and lidar point clouds. A novel Pyramid Fusion Backbone fuses these feature maps at different scales and combines the multimodal features in a feature pyramid to compute valuable multimodal, multi-scale features. The Pyramid Fusion Head aggregates these pyramid features and further refines them in a late fusion step, incorporating the final features of the sensor backbones. The approach is evaluated on two challenging outdoor datasets and different fusion strategies and setups are investigated. It outperforms recent range view based lidar approaches as well as all so far proposed fusion strategies and architectures.

Authors with CRIS profile

Hannah Schieber Juniorprofessur für Human-Centered Computing and Extended Reality

Involved external institutions

Karlsruhe Institute of Technology (KIT) DE Germany (DE) Technische Hochschule Ingolstadt DE Germany (DE)

How to cite

APA:

Schieber, H., Duerr, F., Schoen, T., & Beyerer, J. (2022). Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation. In IEEE Intelligent Vehicles Symposium, Proceedings (pp. 375-381). Aachen, DEU: Institute of Electrical and Electronics Engineers Inc..

MLA:

Schieber, Hannah, et al. "Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation." Proceedings of the 2022 IEEE Intelligent Vehicles Symposium, IV 2022, Aachen, DEU Institute of Electrical and Electronics Engineers Inc., 2022. 375-381.

BibTeX: Download