Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs
Afzal A, Hager G, Wellein G (2020)
Publication Type: Conference contribution
Publication year: 2020
Journal
Publisher: Springer
Book Volume: 12151 LNCS
Pages Range: 391-411
Conference Proceedings Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Event location: Frankfurt
ISBN: 9783030507428
DOI: 10.1007/978-3-030-50743-5_20
Abstract
Analytic, first-principles performance modeling of distributed-memory parallel codes is notoriously imprecise. Even for applications with extremely regular and homogeneous compute-communicate phases, simply adding communication time to computation time does often not yield a satisfactory prediction of parallel runtime due to deviations from the expected simple lockstep pattern caused by system noise, variations in communication time, and inherent load imbalance. In this paper, we highlight the specific cases of provoked and spontaneous desynchronization of memory-bound, bulk-synchronous pure MPI and hybrid MPI+OpenMP programs. Using simple microbenchmarks we observe that although desynchronization can introduce increased waiting time per process, it does not necessarily cause lower resource utilization but can lead to an increase in available bandwidth per core. In case of significant communication overhead, even natural noise can shove the system into a state of automatic overlap of communication and computation, improving the overall time to solution. The saturation point, i.e., the number of processes per memory domain required to achieve full memory bandwidth, is pivotal in the dynamics of this process and the emerging stable wave pattern. We also demonstrate how hybrid MPI-OpenMP programming can prevent desirable desynchronization by eliminating the bandwidth bottleneck among processes. A Chebyshev filter diagonalization application is used to demonstrate some of the observed effects in a realistic setting.
Authors with CRIS profile
Ayesha Afzal
Regionales Rechenzentrum Erlangen (RRZE)
Georg Hager
Regionales Rechenzentrum Erlangen (RRZE)
Gerhard Wellein
Professur für Höchstleistungsrechnen
How to cite
APA:
Afzal, A., Hager, G., & Wellein, G. (2020). Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs. In Ponnuswamy Sadayappan, Bradford L. Chamberlain, Guido Juckeland, Hatem Ltaief (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 391-411). Frankfurt, DE: Springer.
MLA:
Afzal, Ayesha, Georg Hager, and Gerhard Wellein. "Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs." Proceedings of the 35th International Conference on High Performance Computing, ISC High Performance 2020, Frankfurt Ed. Ponnuswamy Sadayappan, Bradford L. Chamberlain, Guido Juckeland, Hatem Ltaief, Springer, 2020. 391-411.
BibTeX: Download