Attention Please: What Transformer Models Really Learn for Process Prediction
Käppel M, Ackermann L, Jablonski S, Härtl S (2024)
Publication Language: English
Publication Type: Conference contribution, Conference Contribution
Publication year: 2024
Publisher: Springer
Series: Lecture Notes in Computer Science
City/Town: Cham
Book Volume: 14940
Pages Range: 203-220
Conference Proceedings Title: Business Process Management 22nd International Conference, BPM 2024, Krakow, Poland, September 1–6, 2024, Proceedings
Event location: Krakau
ISBN: 978-3-031-70395-9
URI: https://link.springer.com/chapter/10.1007/978-3-031-70396-6_12
DOI: 10.1007/978-3-031-70396-6_12
Abstract
Predictive process monitoring aims to support the execution of a process during runtime with various predictions about the further evolution of a process instance. In the last years a plethora of deep learning architectures have been established as state-of-the-art for different prediction targets, among others the transformer architecture. The transformer architecture is equipped with a powerful attention mechanism, assigning attention scores to each input part that allows to prioritize most relevant information leading to more accurate and contextual output. However, deep learning models largely represent a black box, i.e., their reasoning or decision-making process cannot be understood in detail. This paper examines whether the attention scores of a transformer based next-activity prediction model can serve as an explanation for its decision-making. We find that attention scores in next-activity prediction models can serve as explainers and exploit this fact in two proposed graph-based explanation approaches. The gained insights could inspire future work on the improvement of predictive business process models as well as enabling a neural network based mining of process models from event logs.
Authors with CRIS profile
Involved external institutions
Germany (DE)How to cite
APA:
Käppel, M., Ackermann, L., Jablonski, S., & Härtl, S. (2024). Attention Please: What Transformer Models Really Learn for Process Prediction. In Marrella, Andrea Resinas, Manuel Jans, Mieke Rosemann, Michael (Eds.), Business Process Management 22nd International Conference, BPM 2024, Krakow, Poland, September 1–6, 2024, Proceedings (pp. 203-220). Krakau, PL: Cham: Springer.
MLA:
Käppel, Martin, et al. "Attention Please: What Transformer Models Really Learn for Process Prediction." Proceedings of the International Conference on Business Process Management, Krakau Ed. Marrella, Andrea Resinas, Manuel Jans, Mieke Rosemann, Michael, Cham: Springer, 2024. 203-220.
BibTeX: Download