On the strong scaling of the spectral element solver Nek5000 on petascale systems

Offermans N, Marin O, Schanen M, Gong J, Fischer P, Schlatter P (2016)

Publication Type: Conference contribution

Publication year: 2016

Publisher: Association for Computing Machinery

Conference Proceedings Title: ACM International Conference Proceeding Series

Event location: Stockholm, SWE

ISBN: 9781450341226

DOI: 10.1145/2938615.2938617

Abstract

The present work is targeted at performing a strong scaling study of the high-order spectral element uid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent ow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000 - 10; 000 degrees of freedom per core on Mira, 30; 000 - 50; 0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10; 000 and 220; 000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.

Authors with CRIS profile

Philipp Schlatter

Involved external institutions

Royal Institute of Technology / Kungliga Tekniska Högskolan (KTH) SE Sweden (SE) Argonne National Laboratory US United States (USA) (US) University of Illinois at Urbana-Champaign US United States (USA) (US)

How to cite

APA:

Offermans, N., Marin, O., Schanen, M., Gong, J., Fischer, P., & Schlatter, P. (2016). On the strong scaling of the spectral element solver Nek5000 on petascale systems. In ACM International Conference Proceeding Series. Stockholm, SWE: Association for Computing Machinery.

MLA:

Offermans, Nicolas, et al. "On the strong scaling of the spectral element solver Nek5000 on petascale systems." Proceedings of the 2016 Exascale Applications and Software Conference, EASC 2016, Stockholm, SWE Association for Computing Machinery, 2016.

BibTeX: Download