An industrial case study for performance evaluation of hardware-in-the-loop simulators with a combination of network calculus and discrete-event simulation - Funda, Christoph; Funda, Christoph; Konheiser, Tobias; Herpel, Thomas; German, Reinhard; Hielscher, Kai-Steffen (2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering, ICECCME 2022, 2022)
Keywords: hardware-in-the-loop simulation; network calculus; computer performance evaluation; discrete-event simulation

Arrival and Service Curve Measurement-Based Estimation Methods to Analyze and Design Soft Real-Time Streaming Systems with Network Calculus - Funda, Christoph; Marin Garcia, Pablo; German, Reinhard; Hielscher, Kai-Steffen (2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME), 2023)
This paper reviews various methods for estimating arrival and service curves for simple linear functions from experimental data. We experiment with methods found in the literature and discuss the advantages and disadvantages of these methods. We propose two self-developed simple methods, inspired by the literature, to estimate rate-latency service curves that fit our use case to analyze a soft real-time streaming system with network calculus (NC). We also apply three methods to a novel field for performance evaluation with NC: hardware-in-the-loop (HIL) test benches. The performance metrics playback buffer and minimum required pre-buffer time are dimensioned using NC solutions for streaming devices from our previous paper. Linear NC elements are generated in three different ways based on measurements of software processing and network latencies. Playback buffer backlog and delay bounds are calculated based on the three service curve estimation methods using NC. These computed bounds are compared and discussed, and further verified by a discrete-event simulation model with trace-driven input and with distributional fits of processing latencies. The simulation model uses a variation of the workload to prove the system while operating at a soft real-time operating point, with the mean arrival-rate near the mean service-rate and above the minimum service-rate. We recommend using the NC delay metrics with a safety-factor to design the pre-buffer time.<br
Keywords: hardware-in-the-loop simulation; network calculus; discrete-event simulation

Discrete event simulation for the purpose of real-time performance evaluation of distributed hardware-in-the-loop simulators for autonomous driving vehicle validation - Funda, Christoph; German, Reinhard; Hielscher, Kai-Steffen (Electronic Communications of the EASST, 2021)
Hardware-in-the-loop test benches are distributed computer systems including software, hardware and networking devices, which require strict real-time guarantees. To guarantee strict real-time of the simulator the performance needs to be evaluated. To evaluate the timing performance a discrete event simulation model is built up. The input modeling is based on measurements from the real system in a prototype phase. The results of the simulation model are validated with measurements from a prototype of the real system. The workload is increased until the streaming source becomes unstable, by either exceeding a certain limit of bytes or exceeding the number of parallel software processes running on the cores of the central processing unit. To evaluate the performance beyond these limits, the discrete event simulation model needs to be enriched by a scheduler and a hardware model. To provide real-time guarantees an analytical model needs to be built up.
CRIS-Team Scopus Importer:2021-11-26
Keywords: DES; discrete event simulation; distributed system; hardware-in-the-loop simulation; HIL; networks; OMNeT++; real-time performance evaluation

TBASCEM - Tight Bounds with Arrival and Service Curve Estimation by Measurements - Funda, Christoph; Herpel, Thomas; German, Reinhard; Hielscher, Kai-Steffen (ICPE 2024 - Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering, 2024)
This paper aims to solve the challenge of quantifying the performance of Hardware-in-the-Loop (HIL) computer systems used for data re-injection. The system can be represented as a multiple queue and server system that operates on a First-In, First-Out (FIFO) basis. The task at hand involves establishing tight bounds on end-to-end delay and system backlog. This is necessary to optimise buffer and pre-buffer time configurations. Network Calculus (NC) is chosen as the basic analytical framework to achieve this. In the literature, there are different techniques for estimating arrival and service curves from measurement data, which can be used for NC calculations. We have selected four of these methods to be applied to datasets of industrial Timestamp Logging (TL). The problem arises because these conventional methods often produce bounds that are much larger (by a factor of 1000 or more) than the measured maximum values, resulting in inefficient design of HIL system parameters and inefficient resource usage. The proposed approach, called TBASCEM, introduces a reverse engineering approach based on linear NC equations for estimating the parameters of arrival and service curves. By imposing constraints on the equation variables and employing non-linear optimization, TBASCEM searches for a burst parameter estimation which derives tight global delay bounds. In addition, TBASCEM simplifies the run-time measurement process, supporting real-time data acquisition to evaluate and optimise HIL system performance, and enhancing observability to adapt the HIL configuration to new sensor data. The benefits of TBASCEM are clearly that it enables an efficient performance logging of arrival and service curve parameters and with deriving tighter bounds in HIL systems, compared to evaluated state-of-the-art methods, making TBASCEM an invaluable tool for optimising and monitoring streaming applications in non-hard-real-time environments.
CRIS-Team Scopus Importer:2024-05-31
Keywords: Arrival and Service Curve Estimation; Hardware-in-the-loop Test System; Network Calculus; Performance Evaluation and Monitoring; Streaming System