Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Gruber M, Heine M, Oster N, Philippsen M, Fraser G (2023)

Publication Language: English

Publication Type: Conference contribution

Publication year: 2023

Publisher: Institute of Electrical and Electronics Engineers Inc.

Pages Range: 210-221

Conference Proceedings Title: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation, ICST 2023

Event location: Dublin IE

ISBN: 9781665456661

DOI: 10.1109/ICST57152.2023.00028

Abstract

Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

Authors with CRIS profile

Norbert Oster Lehrstuhl für Informatik 2 (Programmiersysteme) Michael Philippsen Lehrstuhl für Informatik 2 (Programmiersysteme)

Related research project(s)

Verification and validation in industrial practice (V&ViP) Jan. 1, 2022 - Jan. 1, 2025

Involved external institutions

BMW AG - Bayerische Motoren Werke / BMW Group DE Germany (DE) Universität Passau DE Germany (DE)

How to cite

APA:

Gruber, M., Heine, M., Oster, N., Philippsen, M., & Fraser, G. (2023). Practical Flaky Test Prediction using Common Code Evolution and Test History Data. In IEEE (Eds.), Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation, ICST 2023 (pp. 210-221). Dublin, IE: Institute of Electrical and Electronics Engineers Inc..

MLA:

Gruber, Martin, et al. "Practical Flaky Test Prediction using Common Code Evolution and Test History Data." Proceedings of the 16th IEEE International Conference on Software Testing, Verification and Validation, ICST 2023, Dublin Ed. IEEE, Institute of Electrical and Electronics Engineers Inc., 2023. 210-221.

BibTeX: Download