Assessing the Performance of Remaining Time Prediction Methods for Business Processes - Roider, Johannes; Nguyen, An; Zanca, Dario; Eskofier, Björn (IEEE Access, 2024)
The prediction of the remaining time for business processes is a major task in predictive process monitoring (PPM). In the last years, various machine learning methods were introduced which reduced error levels steadily. However, the commonly applied metric for optimization and evaluation, the Mean Absolute Error (MAE), has limitations regarding its interpretability. In this work we introduce and evaluate the normalized Mean Absolute Error (nMAE) as an interpretable metric for model evaluation. It accounts for different kinds of label shifts, which are a special type of concept drift that can distort remaining time results. We investigate these concepts in a thorough benchmark study and use them to assess the current state of remaining time prediction for business processes. This includes the evaluation of four different baseline models, identifying the most accurate one. Furthermore, our study compares three different state-of-the-art methods, namely XGBoost, DA-LSTM, and PGT-Net. In contrary to prior studies we find that there is no significant difference in the performance between these models. Additionally, using the nMAE as evaluation metric we find that these models do not perform reasonably well on a range of event logs. Initial ideas for this behaviour are discussed and consolidated along with other findings from the case study into a comprehensive list motivating future research direction
Keywords: Business Process; Graph Neural Network; LSTM; Machine Learning; Predictive Process Monitoring; Process Mining; Remaining Time; XGBoost

Efficient Training of Recurrent Neural Networks for Remaining Time Prediction in Predictive Process Monitoring - Roider, Johannes; Zanca, Dario; Eskofier, Björn (Business Process Management, 22nd International Conference, BPM 2024, Krakow, Poland, September 1–6, 2024, Proceedings, 2024)
An important task in predictive process monitoring is the prediction of the remaining time. The accuracy of methods to solve this task has been steadily improved and verified during the last years. The current state-of-the-art uses Long Short-Term Memory (LSTM) neural networks, represented by the DA-LSTM model. However, training such methods requires substantial amounts of time and memory. This paper addresses specifically these problems. We adjust DA-LSTM and introduce DA-LSTM+ which achieves competitive error levels while reducing the training time significantly. Furthermore, we introduce trace-based sequence encoding as an alternative to prefix encoding, and an approach to use case attributes more efficiently to address time and memory limitations during training. The usage of these is not limited to LSTM’s but they are compatible with any neural network type. We evaluate them together with two alternatives for categorical feature encoding in an extensive benchmark study including eight different model architectures based on DA-LSTM+ and 14 publicly available datasets. The study shows that the training of neural network-based methods can be significantly accelerated with our contributions without affecting model’s performance. Our implementation is memory-efficient and publicly availabl
Keywords: Predictive Process Monitoring; Deep Learning; Process Mining; Remaining Time Prediction

Predictions in Predictive Process Monitoring with Previously Unseen Categorical Values - Roider, Johannes; Wang, Weixin; Zanca, Dario; Matzner, Martin; Eskofier, Björn (Process Mining Workshops, 2024)
Predictive process monitoring (PPM) methods provide users with real-time predictions about ongoing process instances. Machine learning models used for such tasks do not account for data variability, such as the occurrence of previously unseen categorical feature values. Concept drift adaptation solutions are suggested in such scenarios. However, adapting to new feature values requires time and a sample size large enough to train a well-generalizing model. Still, users expect seamless communication during the timeframe between the first occurrence of a new value and the availability of an updated model. Dedicated solutions are needed since encoding techniques like one hot encoding cannot handle previously unseen values by default. In this work, we first introduce and discuss possible solutions from a business perspective, ranging from temporary shutdowns to dedicated manual and technical solutions for an uninterrupted continuation of predictive services. Next, we present five variants for one hot encoding to handle previously unseen categorical values. This is followed by a case study using six real-world event logs and two machine learning models, XGBoost and LSTM, to identify the variants that produce the most reliable remaining time predictions. The study also includes the evaluation of two baseline models as an alternative to the machine learning models. The results show that previously unseen categorical values can be handled on a technical level without severely affecting the remaining time prediction quality. However, future research is required to provide more practical recommendations.<br