Dual-Mode Training with Style Control and Quality Enhancement for Road Image Domain Adaptation - Venator, Moritz; Shen, Fengyi; Aklanoglu, Selcuk; Bruns, Erich; Diepold, Klaus; Maier, Andreas (2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020)
Dealing properly with different viewing conditions remains a key challenge for computer vision in autonomous driving. Domain adaptation has opened new possibilities for data augmentation, translating arbitrary road scene images into different environmental conditions. Although multimodal concepts have demonstrated the capability to separate content and style, we find that existing methods fail to reproduce scenes in the exact appearance given by a reference image. In this paper, we address the aforementioned problem by introducing a style alignment loss between output and reference image. We integrate this concept into a multimodal unsupervised image-to-image translation model with a novel dual-mode training process and additional adversarial losses. Focusing on road scene images, we evaluate our model in various aspects including visual quality and feature matching. Our experiments reveal that we are able to significantly improve both style alignment and image quality in different viewing conditions. Adapting concepts from neural style transfer, our new training approach allows to control the output of multimodal domain adaptation, making it possible to generate arbitrary scenes and viewing conditions for data augmentation.

Enhancing Collaborative Road Scene Reconstruction with Unsupervised Domain Alignment - Venator, Moritz; Aklanoglu, Selcuk; Bruns, Erich; Maier, Andreas (Machine Vision and Applications, 2021)
Scene reconstruction and visual localization in dynamic environments such as street scenes are a challenge due to the lack of distinctive, stable keypoints. While learned convolutional features have proven to be robust to changes in viewing conditions, hand-crafted features still have advantages in distinctiveness and accuracy when applied to Structure from Motion. For collaborative reconstruction of road sections by a car fleet, we apply multimodal domain adaptation as a preprocessing step to align images in their appearance and enhance keypoint matching across viewing conditions while preserving the advantages of hand-crafted features. Training a Generative Adversarial Network for translations between different illumination and weather conditions, we evaluate both qualitative and quantitative aspects of domain adaptation and its impact on feature correspondences. Combined with a multi-feature discriminator, the model is optimized for synthesis of images which do not only improve feature matching but also exhibit a high visual quality. Evaluating the approach with a challenging multi-domain image dataset recorded in various road scenes on multiple test drives, we show that our approach outperforms other traditional and learning-based methods by improving completeness or accuracy of Structure from Motion with multimodal two-domain image collections in eight out of ten test scene
Keywords: Domain adaptation; Image-to-image translation; Structure from Motion; Scene reconstruction; Visual localization

Robust Camera Pose Estimation for Unordered Road Scene Images in Varying Viewing Conditions - Venator, Moritz; Bruns, Erich; Maier, Andreas (IEEE Transactions on Intelligent Vehicles, 2019)
For continuous performance optimization of camera sensor systems in automated driving, training data from rare corner cases occurring in series production cars are required. In this work, we propose collaborative acquisition of camera images via connected car fleets for synthesis of image sequences from arbitrary road sections which are challenging for machine vision. While allowing a scalable hardware architecture inside the cars, this concept demands to reconstruct the recording locations of the individual images aggregated in the back-end. Varying environmental conditions, dynamic scenes, and small numbers of significant landmarks may hamper camera pose estimation through sparse reconstruction from unordered road scene images. Tackling those problems, we extend a state-of-the-art Structure from Motion pipeline by selecting keypoints based on a semantic image segmentation and removing GPS outliers. We present three challenging image datasets recorded on repetitive test drives under differing environmental conditions for evaluation of our method. The results demonstrate that our optimized pipeline is able to reconstruct the camera viewpoints robustly in the majority of road scenes observed while preserving high image registration rates. Reducing the median deviation from GPS measurements by over 48 % for car fleet images, the method increases the accuracy of camera poses dramatically.<br
Keywords: Structure from Motion; 6-DoF camera pose reconstruction; car fleet images; varying environmental conditions

Self-Supervised Learning of Domain-Invariant Local Features for Robust Visual Localization under Challenging Conditions - Venator, Moritz; El Himer, Yassine; Aklanoglu, Selcuk; Bruns, Erich; Maier, Andreas (IEEE Robotics and Automation Letters, 2021)
Visual localization provides the basis for many robotics applications such as autonomous navigation or augmented reality. Especially in outdoor scenes, robust localization requires local features which can be reliably extracted and matched under changing conditions. Previous approaches have applied generative image-to-image translation models to align images in a single domain before correspondence search. In this paper, we invert the concept and elaborate why it is more promising to use image domain adaptation for training of robust local features. Integrating this idea into a self-supervised training framework, we show in various experiments covering image matching, visual localization, and scene reconstruction that our Domain-Invariant SuperPoint (DISP) outperforms existing self-supervised methods in terms of repeatability, generalization, and robustness. In contrast to competitive supervised local features, our modular and fully self-supervised approach can be easily adapted to different domains and localization tasks as it does not require ground truth correspondences for trainin
Keywords: Feature extraction; Location awareness; Training; Visualization; Task analysis; Detectors; Image reconstruction