Kuhnert N, Maier A (2020)
Publication Language: English
Publication Type: Journal article, Online publication
Publication year: 2020
We aim to model an adaptive log file parser. As the content of log files often evolves over time, we established a dynamic statistical model which learns and adapts processing and parsing rules. First, we limit the amount of unstructured text clustering based on semantics of log file lines. Next, we only take the most relevant cluster into account and focus only on those frequent patterns which lead to the desired output table similar to Vaarandi [10]. Furthermore, we transform the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file parsing. With changes in the raw log file distorting learned patterns, we aim the model to adapt automatically in order to maintain high quality output. After training our model on one system type and applying the resulting, adaptive parsing rules to a different system with slightly different log file patterns, we achieve an accuracy over 99.99%.
APA:
Kuhnert, N., & Maier, A. (2020). With Semantics and Hidden Markov Models to an Adaptive Log File Parser. International Journal on Natural Language Computing. https://dx.doi.org/10.5121/ijnlc.2019.8603
MLA:
Kuhnert, Nadine, and Andreas Maier. "With Semantics and Hidden Markov Models to an Adaptive Log File Parser." International Journal on Natural Language Computing (2020).
BibTeX: Download