Efficient Docker Container Management and CI Strategies for Research Software Engineering - Thönnes, Dominik (2024)
Keywords: This presentation focuses on the continuous integration (CI) infrastructure used by two simulation software frameworks with a strong emphasis on performance and scalability: HyTeG; a C++ framework for large-scale high-performance finite element simulations based on geometric multigrid; and waLBerla; a massively parallel framework for multiphysics applications. During this talk; we will detail our approach to managing Docker Containers within our CI environment. As we develop software for research purposes; we often face the challenge of testing various combinations of software packages; especially different compilers. To tackle this; we employ Python scripting and the Spack software manager to create Docker Containers. These containers are then utilized within the CI for our research software. The container creation process is automated within the CI itself; enhancing consistency and manageability. Furthermore; we will explain how Python scripting generates the CI specifications for our GitLab CI. With approximately 150 different jobs in our Pipeline; a generated CI significantly reduces the chances of errors compared to manual creation and maintenance. Moreover; we will shine a light on some of the less conventional uses of CI; such as analyzing build times or conducting code style checks. Our goal for this talk is to share best practices and spread ideas about more efficient CI utilization.