Associating sensor data and reference truth labels: A step towards SOTIF validation of perception sensors - Kryda, Marco; Qiu, Minhao; Berk, Mario; Buschardt, Boris; Antesberger, Tobias; German, Reinhard; Straub, Daniel (Sixth IEEE International Workshop on Automotive Reliability, Test and Safety (ARTS), 2021)
A main requirement for safe automated vehicles is a reliable environment perception in terms of the intended functionality. Perception sensors are intended to detect all surrounding objects. However, uncertainties in the measurement principles and the underlying machine learning algorithms makes it difficult to describe the intended functionality of perception sensors mathematically which is necessary prior to the reliability analysis of those. We approach this problem by, first, applying a multi-object-tracking algorithm and, second, using Density-Based Spatial Clustering of Application with Noise (DBSCAN) to associate tracked objects from multiple sensors. Finally, we transform the clustered object data to a binary format which allows a quantification of the sensor reliabilitie
Keywords: Sensor fusion; Object lists; Reliability Analysis

Exploring the impact of scenario and distance information on the reliability assessment of multi-sensor systems - Qiu, Minhao; Antesberger, Tobias; Bock, Florian; German, Reinhard (2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2022, 2022)
With the growth of self-driving technologies, the reliability analysis of automated driving systems has received considerable attention from both academia and industry. Safety of the intended functionality (SOTIF) serves as one of the primary standards to assure the reliability and safety of the automated driving system. One of its key issues is the performance limitations of perception sensor systems. Generally, the reliability of the perception sensor system depends on the different scenarios of the driving environment. In this work, we investigate the sensor features and dependencies of the front camera and the top LiDAR of the nuTonomy scenes (nuScenes) dataset with respect to scenarios (e.g., rain and night) and distance information (e.g., two distance-based regions of interest). In addition, we apply the obtained parameters to a proven analytical reliability model to examine the impact of scenario and distance information on the reliability assessment.<br
Keywords: SOTIF; Automated driving systems; Reliability analysis; Multi-sensor systems; Discrete-time Markov chains

Improved sensor error definitions for reliability analysis of multi-sensor systems - Qiu, Minhao; Bazan, Peter; Antesberger, Tobias; German, Reinhard (The 7th International Conference on System Reliability and Safety, 2023)
Defining sensor errors is the key issue of the reliability assessment of perception sensor systems. Recently, the most common sensor error types considered are false negatives (FNs) and false positives (FPs). Under certain circumstances, a sensor perception result may trigger a pair of correlated FN and FP, e.g., when detecting an object with inaccurate depth estimation. To investigate the frequency of the occurrence of this type of error, and to provide a more explainable interpretation of the sensor errors, we redefine them into nondetections, ghost objects, and localization errors, where localization errors are caused by low-quality perception results. Furthermore, we propose a three-step association algorithm to identify these error types from perception results and the ground truths, utilizing the Hungarian algorithm with distance matrices based on the Generalized Intersection over Union (GIoU) and Intersection over Union (IoU). Moreover, we extend a proven reliability model to account for the new error types and fusion methods, e.g., Kalman filters. Finally, we conduct a case study on the nuScenes dataset to show the significance of the proposed sensor error definitions and reliability mode
Keywords: Automated driving; SOTIF; Reliability analysis; Perception sensor systems; Sensor error definition

Multi-sensor system simulation based on RESTART algorithm - Qiu, Minhao; Antesberger, Tobias; German, Reinhard (The 67th Annual Reliability and Maintainability Symposium, 2021)
Multi-sensor systems play a central role in modern advanced driver assistant systems. Recently, they become more and more reliable owing to the rapid development of sensor data fusion methods and sensor quality. Since automotive systems are failure critical, suitable descriptions of the safety of multi-sensor systems are required. Their reliability can be determined either with analytical methods or with simulations. In general, modeling with simulations has no constraints in comparison to analytical models and more potential to extend for a complex system. Nevertheless, simulation models can also lead to long run times when they observe infrequent events. Due to the low sensor error rate, it is almost impossible to simulate these sensor system models in practical run time. Therefore, the present study aims to check whether the rare event simulation method Repetitive Simulation Trials After Reaching Thresholds (RESTART) algorithm can make the problem feasible. We employ RESTART simulations to a proven multi-sensor system model, which is on the basis of discrete-time Markov chains and in consideration of dependencies between successive sensor errors and correlation between different sensors. Meanwhile, we compared the results of RESTART simulations with analytical system failure probability. Furthermore, in this study we define a new parameter "acceleration factor" so that we are able to describe the acceleration of RESTART simulations. Based on this factor, we demonstrate the difference between system failure estimation of normal simulation results and related RESTART simulation, which can additionally validate RESTART simulation results. In conclusion, we can approximately accelerate normal simulations ten to a hundred times and at the same time with acceptable accuracy of simulation results.<br
Keywords: Discrete-time Markov chain; Reliability analysis; Multi-sensor system; Rare event simulation; RESTART algorithm

Parameter tuning for a Markov-based multi-sensor system - Qiu, Minhao; Kryda, Marco; Bock, Florian; Antesberger, Tobias; Straub, Daniel; German, Reinhard (2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2021, 2021)
Multi-sensor systems are the key components of automated driving functions. They enhance the quality of the driving experience and assisting in preventing traffic accidents. Due to the rapid evolution of sensor technologies, sensor data collection errors occur rarely. Nonetheless, according to Safety Of The Intended Functionality (SOTIF), an erroneous interpretation of the sensor data can also cause safety hazards. For example the front-camera may not understand the meaning of a traffic sign. Due to safety concerns it is essential to analyze the system reliability throughout the whole development process. In this work, we present an approach to explore the sensor's features, such as the dependencies between successive sensor detection errors and the correlation between different sensors on the KITTI dataset quantitatively. Besides, we apply the learned parameters to a proven multi-sensor system model, which is based on Discrete-time Markov chains, to estimate the reliability of a hypothetical Stereo camera-LiDAR based sensor system.<br

Redundant Sensor-Based Perception Sensor Reliability Estimation from Field Tests without Reference Truth - Kryda, Marco; Qiu, Minhao; Berk, Mario; Buschardt, Boris; Straub, Daniel (SAE Technical Papers, 2023)
The introduction of autonomous vehicles has gained significant attention due to its potential to revolutionize mobility and safety. A critical aspect underpinning the functionality of these autonomous vehicles is their sensor perception system. Demonstrating the reliability of the environment perception sensors and sensor fusion algorithms is, therefore, a necessary step in the development of automated vehicles. Field tests offer testing conditions that come closest to the environment of an automated vehicle in the future. However, a significant challenge in field tests is to obtain a reference truth of the surrounding environment. Here, we propose a pipeline to assess the sensor reliabilities without the need for a reference truth. The pipeline uses a model to estimate the reliability of redundant sensors. To do this, it relies on a binary representation of the surrounding area, which indicates either the presence or absence of an object. Therefore, the pipeline includes another step to convert object lists into this binary representation. Using the pipeline, we estimate the sensor reliabilities from object data derived from the Waymo dataset. Even though we are capable of obtaining close estimates of the sensor reliabilities we find out that the estimation of the sensor reliabilities is not robust for different parameter set
Keywords: Reliability Analysis; Open Loop Testing; SOTIF; Sensor Fusion

Reliability assessment of multi-sensor perception system in automated driving functions - Qiu, Minhao; Bazan, Peter; Antesberger, Tobias; Bock, Florian; German, Reinhard (26th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2021), 2021)
Precise environment perception, which consists of multi-sensor systems, ensures the safety of the automated driving functions (ADFs). With the rapid evolution of sensor technologies, sensor data collection errors occur rarely. Nevertheless, accurate interpretation of the sensor data context, such as 3D multi-object tracking, is still full of challenges. Safety of the Intended Functionality (SOTIF) takes concern of the vulnerability of perception systems. The research of quantitative SOTIF analysis is still ongoing. In this paper, we propose a multi-sensor system model to observe both false negative and false positive errors in different field-of-views. Besides, we also extend a proven Markov-based approach, which takes dependencies between successive sensor errors and correlation between two dependent sensors into account, to model three correlated sensors sharing the same region of interest. In the end, we present a numerical example to illustrate the quantitative reliability analysis of a multi-sensor perception system according to SOTIF.<br
Keywords: Multi-sensor systems; Automated driving functions; SOTIF; Reliability analysis; Discrete-time Markov chains