Addressing White-box Modeling and Simulation Challenges in Parallel Computing - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (SIGSIM-PADS '22: SIGSIM Conference on Principles of Advanced Discrete Simulation, 2022)

A domain-specific language and matrix-free stencil code for investigating electronic properties of Dirac and topological materials - Pieper, Andreas; Hager, Georg; Fehske, Holger (International Journal of High Performance Computing Applications, 2020)
We introduce PVSC-DTM (Parallel Vectorized Stencil Code for Dirac and Topological Materials), a library and code generator based on a domain-specific language tailored to implement the specific stencil-like algorithms that can describe Dirac and topological materials such as graphene and topological insulators in a matrix-free way. The generated hybrid-parallel (MPI+OpenMP) code is fully vectorized using Single Instruction Multiple Data (SIMD) extensions. It is significantly faster than matrix-based approaches on the node level and performs in accordance with the roofline model. We demonstrate the chip-level performance and distributed-memory scalability of basic building blocks such as sparse matrix-(multiple-) vector multiplication on modern multicore CPUs. As an application example, we use the PVSC-DTM scheme to (i) explore the scattering of a Dirac wave on an array of gate-defined quantum dots, to (ii) calculate a bunch of interior eigenvalues for strong topological insulators, and to (iii) discuss the photoemission spectra of a disordered Weyl semimetal.
CRIS-Team Scopus Importer:2020-10-02
Keywords: domain-specific language; Stencil code; topological quantum matter

A Flexible Patch-Based Lattice Boltzmann Parallelization Approach for Heterogeneous GPU-CPU Clusters - Feichtinger, Christian; Habich, Johannes; Köstler, Harald; Hager, Georg; Rüde, Ulrich; Wellein, Gerhard (2010)
UnivIS-Import:2016-06-30:Pub.2010.tech.IMMD.lsinfs.aflexi

A flexible Patch-based lattice Boltzmann parallelization approach for heterogeneous GPU-CPU clusters - Feichtinger, Christian; Köstler, Harald; Hager, Georg; Rüde, Ulrich; Wellein, Gerhard; Habich, Johannes (Parallel Computing, 2011)
Sustaining a large fraction of single GPU performance in parallel computations is considered to be the major problem of GPU-based clusters. We address this issue in the context of a lattice Boltzmann flow solver that is integrated in the WaLBerla software framework. Our multi-GPU implementation uses a block-structured MPI parallelization and is suitable for load balancing and heterogeneous computations on CPUs and GPUs. The overhead required for multi-GPU simulations is discussed in detail. It is demonstrated that a large fraction of the kernel performance can be sustained for weak scaling on InfiniBand clusters, leading to excellent parallel efficiency. However, in strong scaling scenarios using multiple GPUs is much less efficient than running CPU-only simulations on IBM BG/P and x86-based clusters. Hence, a cost analysis must determine the best course of action for a particular simulation task and hardware configuration. Finally we present weak scaling results of heterogeneous simulations conducted on CPUs and GPUs simultaneously, using clusters equipped with varying node configurations. © 2011 Elsevier B.V. All rights reserved.
UnivIS-Import:2015-03-09:Pub.2011.tech.IMMD.lsinfs.aflexi
Keywords: CUDA; Heterogeneous computations; Lattice Boltzmann method; MPI

Algebraic temporal blocking for sparse iterative solvers on multi-core CPUs - Alappat, Christie; Thies, Jonas; Hager, Georg; Fehske, Holger; Wellein, Gerhard (International Journal of High Performance Computing Applications, 2024)
Sparse linear iterative solvers are essential for many large-scale simulations. Much of the runtime of these solvers is often spent in the implicit evaluation of matrix polynomials via a sequence of sparse matrix-vector products. A variety of approaches has been proposed to make these polynomial evaluations explicit (i.e., fix the coefficients), e.g., polynomial preconditioners or s-step Krylov methods. Furthermore, it is nowadays a popular practice to approximate triangular solves by a matrix polynomial to increase parallelism. Such algorithms allow to evaluate the polynomial using a so-called matrix power kernel (MPK), which computes the product between a power of a sparse matrix (Formula presented.) and a dense vector (Formula presented.), i.e., (Formula presented.), or a related operation. Recently we have shown that using the level-based formulation of sparse matrix-vector multiplications in the Recursive Algebraic Coloring Engine (RACE) framework we can perform temporal cache blocking of MPK to increase its performance. In this work, we demonstrate the application of this cache-blocking optimization in sparse iterative solvers. By integrating the RACE library into the Trilinos framework, we demonstrate the speedups achieved in (preconditioned) s-step GMRES, polynomial preconditioners, and algebraic multigrid (AMG). For MPK-dominated algorithms we achieve speedups of up to 3 (Formula presented.) on modern multi-core compute nodes. For algorithms with moderate contributions from subspace orthogonalization, the gain reduces significantly, which is often caused by the insufficient quality of the orthogonalization routines. Finally, we showcase the application of RACE-accelerated solvers in a real-world wind turbine simulation (Nalu-Wind) and highlight the new opportunities and perspectives opened up by RACE as a cache-blocking technique for MPK-enabled sparse solvers.
CRIS-Team Scopus Importer:2024-10-11
Keywords: cache blocking; iterative solvers; matrix polynomial; performance; Sparse matrices

Analysis of intel’s haswell microarchitecture using the ECM model and microbenchmarks - Hofmann, Johannes; Fey, Dietmar; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Springer Verlag, 2016)
Keywords: Architecture analysis; Intel Haswell; ECM model; Performance modeling

Analysis of Intel's Haswell Microarchitecture Using the ECM Model and Microbenchmarks - Hofmann, Johannes; Fey, Dietmar; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Architecture of Computing Systems – ARCS 2016: 29th International Conference, Nuremberg, Germany, April 4-7, 2016, Proceedings, 2016)
UnivIS-Import:2017-01-09:Pub.2016.tech.IMMD.IMMD3.analys

Analytical performance estimation during code generation on modern GPUs - Ernst, Dominik; Holzer, Markus; Hager, Georg; Knorr, Matthias; Wellein, Gerhard (Journal of Parallel and Distributed Computing, 2023)
Automatic code generation is frequently used to create implementations of algorithms specifically tuned to particular hardware and application parameters. The code generation process involves the selection of adequate code transformations, tuning parameters, and parallelization strategies. We propose an alternative to time-intensive autotuning, scenario-specific performance models, or black-box machine learning to select the best-performing configuration. This paper identifies the relevant performance-defining mechanisms for memory-intensive GPU applications through a performance model coupled with an analytic hardware metric estimator. This enables a quick exploration of large configuration spaces to identify highly efficient code candidates with high accuracy. We examine the changes of the A100 GPU architecture compared to the predecessor V100 and address the challenges of how to model the data transfer volumes through the new memory hierarchy. We show how our method can be coupled to the “pystencils” stencil code generator, which is used to generate kernels for a range-four 3D-25pt stencil and a complex two-phase fluid solver based on the Lattice Boltzmann Method. For both, it delivers a ranking that can be used to select the best-performing candidate. The method is not limited to stencil kernels but can be integrated into any code generator that can generate the required address expressions.
CRIS-Team Scopus Importer:2022-12-23
Keywords: Analytical performance modeling; GPU; GPU performance model; Layer condition; Stencil codes

Analytic Modeling of Idle Waves in Parallel Programs: Communication, Cluster Topology, and Noise Impact - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2021)
Most distributed-memory bulk-synchronous parallel programs in HPC assume that compute resources are available continuously and homogeneously across the allocated set of compute nodes. However, long one-off delays on individual processes can cause global disturbances, so-called idle waves, by rippling through the system. This process is mainly governed by the communication topology of the underlying parallel code. This paper makes significant contributions to the understanding of idle wave dynamics. We study the propagation mechanisms of idle waves across the processes of MPI-parallel programs. We present a validated analytic model for their propagation velocity with respect to communication parameters and topology, with a special emphasis on sparse communication patterns. We study the interaction of idle waves with MPI collectives and show that, depending on the implementation, a collective may be permeable to the wave. Finally we analyze two mechanisms of idle wave decay: topological decay, which is rooted in differences in communication characteristics among parts of the system, and noise-induced decay, which is caused by system or application noise. We show that noise-induced decay is largely independent of noise characteristics but depends only on the overall noise power. An analytic expression for idle wave decay rate with respect to noise power is derived. For model validation we use microbenchmarks and stencil algorithms on three different supercomputing platforms.
CRIS-Team Scopus Importer:2021-08-20

Analytic performance model for parallel overlapping memory-bound kernels - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (Concurrency and Computation-Practice & Experience, 2022)
Complex applications running on multicore processors show a rich performance phenomenology. The growing number of cores per ccNUMA domain complicates performance analysis of memory-bound code since system noise, load imbalance, or task-based programming models can lead to thread desynchronization. Hence, the simplifying assumption that all cores execute the same loop can not be upheld. Motivated by observations on plain and modified versions of the HPCG benchmark, we construct a performance model of execution of memory-bound loop kernels. It can predict the memory bandwidth share per kernel on a memory contention domain depending on the number of active cores and which other workload the kernel is paired with. The only code features required are the single-thread memory request fraction per kernel, which is directly related to the single-thread memory bandwidth, and its saturated bandwidth. The former can either be measured directly or predicted using the Execution-Cache-Memory performance model. The computational intensity of the kernels and the detailed structure of the code is of no significance. We validate our model on Intel Broadwell, Intel Cascade Lake, and AMD Rome processors pairing various streaming and stencil kernels. The error in predicting the bandwidth share per kernel is less than 8%.<br

Analytic performance modeling and analysis of detailed neuron simulations - Cremonesi, Francesco; Hager, Georg; Wellein, Gerhard; Schuermann, Felix (International Journal of High Performance Computing Applications, 2020)
Big science initiatives are trying to reconstruct and model the brain by attempting to simulate brain tissue at larger scales and with increasingly more biological detail than previously thought possible. The exponential growth of parallel computer performance has been supporting these developments, and at the same time maintainers of neuroscientific simulation code have strived to optimally and efficiently exploit new hardware features. Current state-of-the-art software for the simulation of biological networks has so far been developed using performance engineering practices, but a thorough analysis and modeling of the computational and performance characteristics, especially in the case of morphologically detailed neuron simulations, is lacking. Other computational sciences have successfully used analytic performance engineering, which is based on "white-box," that is, first-principles performance models, to gain insight on the computational properties of simulation kernels, aid developers in performance optimizations and eventually drive codesign efforts, but to our knowledge a model-based performance analysis of neuron simulations has not yet been conducted. We present a detailed study of the shared-memory performance of morphologically detailed neuron simulations based on the Execution-Cache-Memory performance model. We demonstrate that this model can deliver accurate predictions of the runtime of almost all the kernels that constitute the neuron models under investigation. The gained insight is used to identify the main governing mechanisms underlying performance bottlenecks in the simulation. The implications of this analysis on the optimization of neural simulation software and eventually codesign of future hardware architectures are discussed. In this sense, our work represents a valuable conceptual and quantitative contribution to understanding the performance properties of biological networks simulations.
CRIS-Team WoS Importer:2020-04-21

An Analysis of Core- and Chip-Level Architectural Features in Four Generations of Intel Server Processors - Hofmann, Johannes; Hager, Georg; Wellein, Gerhard; Fey, Dietmar (High Performance Computing. ISC 2017. Lecture Notes in Computer Science, vol 10266, 2017)
UnivIS-Import:2017-07-10:Pub.2017.tech.IMMD.IMMD3.ananal

An Evaluation of Different I/O Techniques for Checkpoint/Restart - Shahzad, Faisal; Wittmann, Markus; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Parallel and Distributed Processing Symposium Workshops PhD Forum (IPDPSW), 2013 IEEE 27th International, 2013)
Today's High Performance Computing (HPC) clusters consist of hundreds of thousands of CPUs, memory units, complex networks, and other components. Such an extreme level of hardware parallelism reduces the mean time to failure (MTTF) of the overall cluster. The future of HPC urgently demands to develop environments that facilitate programs to run successfully even in the presence of failures. Checkpoint/Restart (C/R) is one of the most common techniques to provide fault tolerance. C/R is relatively easy to implement, but typically it introduces significant overhead in the runtime of the application. In this paper, a check pointing technique is presented that significantly reduces the checkpoint overhead and is highly scalable. This is achieved by overlapping the I/O for writing the checkpoint with the computation of the application. For this asynchronous check pointing technique, a theoretical model is developed to estimate the checkpoint overhead. An implementation of this technique is then benchmarked and compared with other check pointing strategies. We show our approach to have marginal overhead as opposite to standard synchronous check pointing for typical application scenarios. A comparison with the node-level check pointing technique by using Scalable Checkpoint/Restart (SCR) library is also presented. © 2013 IEEE.
UnivIS-Import:2015-04-16:Pub.2013.tech.IMMD.phleir.aneval
Keywords: fault tolerance; asynchronous checkpointing; multi-stage checkpointing; checkpoint/restart; MPI

A Recursive Algebraic Coloring Technique for Hardware-efficient Symmetric Sparse Matrix-vector Multiplication - Alappat, Christie; Basermann, Achim; Bishop, Alan R.; Fehske, Holger; Hager, Georg; Schenk, Olaf; Thies, Jonas; Wellein, Gerhard (ACM Transactions on Parallel Computing, 2020)
The symmetric sparse matrix-vector multiplication (SymmSpMV) is an important building block for many numerical linear algebra kernel operations or graph traversal applications. Parallelizing SymmSpMV on today's multicore platforms with up to 100 cores is difficult due to the need to manage conflicting updates on the result vector. Coloring approaches can be used to solve this problem without data duplication, but existing coloring algorithms do not take load balancing and deep memory hierarchies into account, hampering scalability and full-chip performance. In this work, we propose the recursive algebraic coloring engine (RACE), a novel coloring algorithm and open-source library implementation that eliminates the shortcomings of previous coloring methods in terms of hardware efficiency and parallelization overhead. We describe the level construction, distance-k coloring, and load balancing steps in RACE, use it to parallelize SymmSpMV, and compare its performance on 31 sparse matrices with other state-of-the-art coloring techniques and Intel MKL on two modern multicore processors. RACE outperforms all other approaches substantially. By means of a parameterized roofline model, we analyze the SymmSpMV performance in detail and discuss outliers. While we focus on SymmSpMV in this article, our algorithm and software are applicable to any sparse matrix operation with data dependencies that can be resolved by distance-k coloring.
CRIS-Team Scopus Importer:2020-10-30
Keywords: graph algorithms; graph coloring; memory hierarchies; scheduling; Sparse matrix; sparse symmetric matrix-vector multiplication

A survey of checkpoint/restart techniques on distributed memory systems - Shahzad, Faisal; Wittmann, Markus; Kreutzer, Moritz; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Parallel Processing Letters, 2013)
The road to exascale computing poses many challenges for the High Performance Computing (HPC) community. Each step on the exascale path is mainly the result of a higher level of parallelism of the basic building blocks (i.e., CPUs, memory units, networking components, etc.). The reliability of each of these basic components does not increase at the same rate as the rate of hardware parallelism. This results in a reduction of the mean time to failure (MTTF) of the whole system. A fault tolerance environment is thus indispensable to run large applications on such clusters. Checkpoint/Restart (C/R) is the classic and most popular method to minimize failure damage. Its ease of implementation makes it useful, but typically it introduces significant overhead to the application. Several efforts have been made to reduce the C/R overhead. In this paper we compare various C/R techniques for their overheads by implementing them on two different categories of applications. These approaches are based on parallel-file-system (PFS)-level checkpoints (synchronous/ asynchronous) and node-level checkpoints. We utilize the Scalable Checkpoint/Restart (SCR) library for the comparison of node-level checkpoints. For asynchronous PFS-level checkpoints, we use the Damaris library, the SCR asynchronous feature, and application-based checkpointing via dedicated threads. Our baseline for overhead comparison is the naïve application-based synchronous PFS-level checkpointing method. A 3D lattice-Boltzmann (LBM) flow solver and a Lanczos eigenvalue solver are used as prototypical applications in which all the techniques considered here may be applied. © 2013 World Scientific Publishing Company.
UnivIS-Import:2015-03-09:Pub.2013.tech.IMMD.phleir.asurve
Keywords: checkpoint/restart; Fault tolerance; multi-stage checkpointing; synchronous vs. asynchronous checkpointing

A unified sparse matrix data format for efficient general sparse matrix-vector multiplication on modern processors with wide SIMD units - Kreutzer, Moritz; Hager, Georg; Wellein, Gerhard; Fehske, Holger; Bishop, Alan R. (SIAM Journal on Scientific Computing, 2014)
Sparse matrix-vector multiplication (spMVM) is the most time-consuming kernel in many numerical algorithms and has been studied extensively on all modern processor and accelerator architectures. However, the optimal sparse matrix data storage format is highly hardware-specific, which could become an obstacle when using heterogeneous systems. Also, it is as yet unclear how the wide single instruction multiple data (SIMD) units in current multi- and many-core processors should be used most efficiently if there is no structure in the sparsity pattern of the matrix.We suggest SELLC- s, a variant of Sliced ELLPACK, as a SIMD-friendly data format which combines long-standing ideas from general-purpose graphics processing units and vector computer programming. We discuss the advantages of SELL-C-s compared to established formats like Compressed Row Storage and ELLPACK and show its suitability on a variety of hardware platforms (Intel Sandy Bridge, Intel Xeon Phi, and Nvidia Tesla K20) for a wide range of test matrices from different application areas. Using appropriate performance models we develop deep insight into the data transfer properties of the SELL-C-s spMVM kernel. SELL-C-s comes with two tuning parameters whose performance impact across the range of test matrices is studied and for which reasonable choices are proposed. This leads to a hardware-independent ("catch-all") sparse matrix format, which achieves very high efficiency for all test matrices across all hardware platforms.
UnivIS-Import:2015-03-09:Pub.2014.tech.IMMD.phleir.aunifi
Keywords: Data format; Performance model; SIMD; Sparse matrix; Sparse matrix-vector multiplication

Automated Instruction Stream Throughput Prediction for Intel and AMD Microarchitectures - Laukemann, Jan; Hammer, Julian; Hofmann, Johannes; Hager, Georg; Wellein, Gerhard (2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), 2018)
An accurate prediction of scheduling and execution of instruction streams is a necessary prerequisite for predicting the in-core performance behavior of throughput-bound loop kernels on out-of-order processor architectures. Such predictions are an indispensable component of analytical performance models, such as the Roofline and the Execution-Cache-Memory (ECM) model, and allow a deep understanding of the performance-relevant interactions between hardware architecture and loop code. We present the Open Source Architecture Code Analyzer (OSACA), a static analysis tool for predicting the execution time of sequential loops comprising x86 instructions under the assumption of an infinite first-level cache and perfect out-of-order scheduling. We show the process of building a machine model from available documentation and semi-automatic benchmarking, and carry it out for the latest Intel Skylake and AMD Zen micro-architectures. To validate the constructed models, we apply them to several assembly kernels and compare runtime predictions with actual measurements. Finally we give an outlook on how the method may be generalized to new architectures.<br
Keywords: Out of order; Computer architecture; Analytical models; Computational modeling; Predictive models; Runtime; Benchmark testing; benchmarking; performance modeling; performance engineering; architecture analysis; static analysis

Automated instruction stream throughput prediction for intel and AMD microarchitectures - Laukemann, Jan; Hammer, Julian; Hofmann, Johannes; Hager, Georg; Wellein, Gerhard (Proceedings of PMBS 2018: Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2019)
An accurate prediction of scheduling and execution of instruction streams is a necessary prerequisite for predicting the in-core performance behavior of throughput-bound loop kernels on out-of-order processor architectures. Such predictions are an indispensable component of analytical performance models, such as the Roofline and the Execution-Cache-Memory (ECM) model, and allow a deep understanding of the performance-relevant interactions between hardware architecture and loop code. We present the Open Source Architecture Code Analyzer (OSACA), a static analysis tool for predicting the execution time of sequential loops comprising x86 instructions under the assumption of an infinite first-level cache and perfect out-of-order scheduling. We show the process of building a machine model from available documentation and semi-automatic benchmarking, and carry it out for the latest Intel Skylake and AMD Zen micro-architectures. To validate the constructed models, we apply them to several assembly kernels and compare runtime predictions with actual measurements. Finally we give an outlook on how the method may be generalized to new architectures.
Created from Fastlane, Scopus look-up
Keywords: architecture analysis; benchmarking; performance engineering; performance modeling; static analysis

Automatic loop kernel analysis and performance modeling with kerncraft - Hammer, Julian; Hager, Georg; Eitzinger, Jan; Wellein, Gerhard (6th International Workshop in Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, PMBS 2015 - Held as part of the 27th ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, 2015)

Automatic Loop Kernel Analysis and Performance Modeling With Kerncraft - Hammer, Julian; Hager, Georg; Eitzinger, Jan; Wellein, Gerhard (Proceedings of the 6th International Workshop on Performance Modeling, Benchmarking, and Simulation of High Performance Computing Systems, 2015)
UnivIS-Import:2017-12-18:Pub.2015.tech.IMMD.phleir.automa

Automatic Throughput and Critical Path Analysis of x86 and ARM Assembly Kernels - Laukemann, Jan; Hammer, Julian; Hager, Georg; Wellein, Gerhard (10th IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, PMBS 2019, 2019)
Keywords: benchmarking; architecture analysis; performance modeling; performance engineering; static analysis

Benchmark analysis and application results for lattice Boltzmann simulations on NEC SX vector and Intel Nehalem systems - Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Parallel Processing Letters, 2009)
Classic vector systems have all but vanished from recent TOP500 lists. Looking at the recently introduced NEC SX-9 series, we benchmark its memory subsystem using the low level vector triad and employ the kernel of an advanced lattice Boltzmann flow solver to demonstrate that classic vectors still combine excellent performance with a well-established optimization approach. To investigate the multi-node performance, the flow field in a real porous medium is simulated using the hybrid MPI/OpenMP parallel ILBDC lattice Boltzmann application code. Results for a commodity Intel Nehalem-based cluster are provided for comparison. Clusters can keep up with the vector systems, however, require massive parallelism and thus much more effort to provide a good domain decomposition. © 2009 World Scientific Publishing Company.
UnivIS-Import:2015-03-09:Pub.2009.tech.IMMD.phleir._bench
Keywords: Hybrid parallel; Intel Nehalem; Lattice Boltzmann flow solver; NEC SX-9; Sustained performance

Bridging the architecture gap: Abstracting performance-relevant properties of modern server processors - Hofmann, Johannes; Alappat, Christie; Hager, Georg; Fey, Dietmar; Wellein, Gerhard (Supercomputing Frontiers and Innovations, 2020)
We propose several improvements to the execution-cache-memory (ECM) model, an analytic performance model for predicting single-and multicore runtime of steady-state loops on server processors. The model is made more general by strictly differentiating between application and machine models: an application model comprises the loop code, problem sizes, and other runtime parameters, while a machine model is an abstraction of all performance-relevant properties of a processor. Moreover, new first principles underlying the model's estimates are derived from common microarchitectural features implemented by today's server processors to make the model more architecture independent, thereby extending its applicability beyond Intel processors. We introduce a generic method for determining machine models, and present results for relevant server-processor architectures by Intel, AMD, IBM, and Marvell/Cavium. Considering this wide range of architectures, the set of features required for adequate performance modeling is surprisingly small. To validate our approach, we compare performance predictions to empirical data for an OpenMP-parallel preconditioned CG algorithm, which includes compute-and memory-bound kernels. Both single-and multicore analysis shows that the model exhibits average and maximum relative errors of 5 % and 10 %. Deviations from the model and insights gained are discussed in detail.
CRIS-Team Scopus Importer:2020-08-07
Keywords: AMD; Analytic modeling; ARM; Execution-cache-memory model; IBM; Intel; Microarchitecture comparison; Performance evaluation; Performance modeling

Building a Fault Tolerant Application Using the GASPI Communication Layer - Shahzad, Faisal; Kreutzer, Moritz; Zeiser, Thomas; Machado, Rui; Pieper, Andreas; Hager, Georg; Wellein, Gerhard (Proceedings of FTS 2015, 2015)
UnivIS-Import:2016-06-01:Pub.2015.tech.IMMD.phleir.buildi
Keywords: Checkpoint-restart; Fault detection; Fault recovery; Fault tolerance; GASPI; GPI; Pre-allocated spare processes

Challenges and Potentials of Emerging Multicore Architectures - Stürmer, Markus; Wellein, Gerhard; Hager, Georg; Köstler, Harald; Rüde, Ulrich (High Performance Computing in Science and Engineering Garching-Munich 2007, 2009)
We present performance results on two current multicore architectures, a STI (Sony, Toshiba, and IBM) Cell processor included in the new Playstation™ 3 and a Sun UltraSPARC T2 ("Niagara 2") machine. On the Niagara 2 we analyze typical performance patterns that emerge from the peculiar way the memory controllers are activated on this chip using the standard STREAM benchmark and a shared-memory parallel lattice Boltzmann code. On the Cell processor we measure the memory bandwidth and run performance tests for LBM simulations. Additionally, we show results for an application in image processing on the Cell processor, where it is required to solve nonlinear anisotropic PDEs.
UnivIS-Import:2015-04-16:Pub.2009.tech.IMMD.lsinfs.challe_4

Chebyshev filter diagonalization on modern manycore processors and GPGPUs - Kreutzer, Moritz; Ernst, Dominik; Bishop, Alan R.; Fehske, Holger; Hager, Georg; Nakajima, K.; Wellein, Gerhard (Springer Verlag, 2018)

Chip-level and multi-node analysis of energy-optimized lattice Boltzmann CFD simulations - Wittmann, Markus; Hager, Georg; Zeiser, Thomas; Treibig, Jan; Wellein, Gerhard (Concurrency and Computation-Practice & Experience, 2015)
UnivIS-Import:2016-06-01:Pub.2015.tech.IMMD.phleir.chiple

CloverLeaf on Intel Multi-Core CPUs: A Case Study in Write-Allocate Evasion - Laukemann, Jan; Gruber, Thomas; Hager, Georg; Oryspayev, Dossay; Wellein, Gerhard (2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2024)
In this paper we analyze the MPI-only version of the CloverLeaf code from the SPEChpc 2021 benchmark suite on recent Intel Xeon "Ice Lake" and "Sapphire Rapids" server CPUs. We observe peculiar breakdowns in performance when the number of processes is prime. Investigating this effect, we create first-principles data traffic models for each of the stencil-like hotspot loops. With application measurements and microbenchmarks to study memory data traffic behavior, we can connect the breakdowns to SpecI2M, a new write-allocate evasion feature in current Intel CPUs. For serial and full-node cases we are able to predict the memory data volume analytically with an error of a few percent. We find that if the number of processes is prime, SpecI2M fails to work properly, which we can attribute to short inner loops emerging from the one-dimensional domain decomposition in this case. We can also rule out other possible causes of the prime number effect, such as breaking layer conditions, MPI communication overhead, and load imbalance.<br
CRIS-Team Scopus Importer:2024-07-26
Keywords: Sapphire Rapids; SPEChpc 2021; Ice Lake; non-temporal stores; SpecI2M; write allocate; CloverLeaf

Collecting and presenting reproducible intranode stencil performance: INSPECT - Hornich, Julian; Hammer, Julian; Hager, Georg; Gruber, Thomas; Wellein, Gerhard (Supercomputing Frontiers and Innovations, 2019)
Stencil algorithms have been receiving considerable interest in HPC research for decades. The techniques used to approach multi-core stencil performance modeling and engineering span basic runtime measurements, elaborate performance models, detailed hardware counter analysis, and thorough scaling behavior evaluation. Due to the plurality of approaches and stencil patterns, we set out to develop a generalizable methodology for reproducible measurements accompanied by state-of-the-art performance models. Our open-source toolchain and collected results are publicly available in the "Intranode Stencil Performance Evaluation Collection" (INSPECT). We present the underlying methods, models and tools involved in gathering and documenting the performance behavior of a collection of typical stencil patterns across multiple architectures and hardware configuration options. Our aim is to endow performance-aware application developers with reproducible baseline performance data and validated models to initiate a well-defined process of performance assessment and optimization. All data is available for inspection: source code, produced assembly, performance measurements, hardware performance counter data, single-core and multicore Roofline and ECM (execution-cache-memory) performance models, and machine properties. Deviations between measured performance and performance models become immediately evident and can be investigated. We also give hints as to how INSPECT can be used in practice for custom code analysis.
CRIS-Team Scopus Importer:2021-04-20
Keywords: Cache effects; Ecm; Memory hierarchy; Multi-core; Performance analysis; Performance modeling; Roofline; Single-node; Stencils

Comparing the Performance of Different x86 SIMD Instruction Sets for a Medical Imaging Application on Modern Multi- and Manycore Chips - Hofmann, Johannes; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Proceedings of the 2014 Workshop on Programming models for SIMD/Vector processing, 2014)
Single Instruction, Multiple Data (SIMD) vectorization is a major driver of performance in current architectures, and is mandatory for achieving good performance with codes that are limited by instruction throughput. We investigate the efficiency of different SIMDvectorized implementations of the RabbitCT benchmark. RabbitCT performs 3D image reconstruction by back projection, a vital operation in computed tomography applications. The underlying algorithm is a challenge for vectorization because it consists, apart from a streaming part, also of a bilinear interpolation requiring scattered access to image data.We analyze the performance of SSE (128 bit), AVX (256 bit), AVX2 (256 bit), and IMCI (512 bit) implementations on recent Intel x86 systems. A special emphasis is put on the vector gather implementation on Intel Haswell and Knights Corner microarchitectures. Finally we discuss why GPU implementations perform much better for this specific algorithm. Copyright © 2014 ACM.
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD3._compa
Keywords: Back projection; Computed tomography; Gather; Intel MIC; Performance; SIMD

Comparison of Different Propagation Steps for Lattice Boltzmann Methods - Wittmann, Markus; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Computers & Mathematics with Applications, 2013)
UnivIS-Import:2015-03-09:Pub.2012.tech.IMMD.phleir._compa

Complexities of Performance Prediction for Bandwidth-Limited Loop Kernels on Multi-Core Architectures - Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (High Performance Computing in Science and Engineering, Garching/Munich 2009, 2010)
UnivIS-Import:2015-04-16:Pub.2010.tech.IMMD.phleir._compl

Core-Level Performance Engineering with the Open-Source Architecture Code Analyzer (OSACA) and the Compiler Explorer - Laukemann, Jan; Hager, Georg (ICPE 2023 - Companion of the 2023 ACM/SPEC International Conference on Performance Engineering, 2023)
While many developers put a lot of effort into optimizing large-scale parallelism, they often neglect the importance of an efficient serial code. Even worse, slow serial code tends to scale very well, hiding the fact that resources are wasted because no definite hardware performance limit ("bottleneck") is exhausted. This tutorial conveys the required knowledge to develop a thorough understanding of the interactions between software and hardware on the level of a single CPU core and the lowest memory hierarchy level (the L1 cache). We introduce general out-of-order core architectures and their typical performance bottlenecks using modern x86-64 (Intel Ice Lake) and ARM (Fujitsu A64FX) processors as examples. We then go into detail about x86 and AArch64 assembly code, specifically including vectorization (SIMD), pipeline utilization, critical paths, and loop-carried dependencies. We also demonstrate performance analysis and performance engineering using the Open-Source Architecture Code Analyzer (OSACA) in combination with a dedicated instance of the well-known Compiler Explorer. Various hands-on exercises allow attendees to make their own experiments and measurements and identify in-core performance bottlenecks. Furthermore, we show real-life use cases to emphasize how profitable in-core performance engineering can be.
CRIS-Team Scopus Importer:2023-05-26
Keywords: in-core; performance engineering; performance modeling

CRAFT: A library for easier application-level Checkpoint/Restart and Automatic Fault Tolerance - Shahzad, Faisal; Thies, Jonas; Kreutzer, Moritz; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (IEEE Transactions on Parallel and Distributed Systems, 2018)
Keywords: Kernel; Libraries; Fault tolerant systems; Hardware; User-Level Failure Mitigation (ULFM; automatic fault tolerance; Application-level checkpoint/restart; Fault tolerance; Checkpointing

cxHPC: Setting up ByGRID — First Steps Towards an e-Science Infrastructure in Bavaria - Hager, Georg; Zeiser, Thomas; Heller, Helmut (High Performance Computing in Science and Engineering, Garching 2004, 2004)
UnivIS-Import:2018-09-06:Pub.2004.zwiss.rrze.zensys.cxhpcs

cxHPC: Setting up ByGRID First Steps Towards an e-Science Infrastructure in Bavaria - Hager, Georg; Zeiser, Thomas; Heller, Helmut (High Performance Computing in Science and Engineering, Garching 2004: Transactions of the KONWIHR Result Workshop, October 1415, 2004, Technical University of Munich, Garching, Germany, 2005)
UnivIS-Import:2015-04-20:Pub.2005.zwiss.rrze.zensys.cxhpcs

Data access characteristics and optimizations for SUN ULTRASPARC T2 AND T2+ systems - Hager, Georg; Zeiser, Thomas; Wellein, Gerhard (Parallel Processing Letters, 2008)
Processor and system architectures that feature multiple memory controllers and/or ccNUMA characteristics are prone to show bottlenecks and erratic performance numbers on scientific codes. Although cache thrashing, aliasing conflicts, and ccNUMA locality and contention problems are well known for many types of systems, they take on peculiar forms on the new Sun UltraSPARC T2 and T2+ processors, which we use here as prototypical multi-core designs. We analyze performance patterns in low-level and application benchmarks and put some emphasis on a comparison of performance features between T2 and its successor. Furthermore we show ways to circumvent bottlenecks by careful data layout, placement and padding. © 2008 World Scientific Publishing Company.
UnivIS-Import:2015-03-09:Pub.2008.zwiss.rrze.zensys.dataac
Keywords: CcNUMA; Data layout; Memory access; Memory controllers; Multi-core

Data access optimizations for highly threaded multi-core CPUs with multiple memory controllers - Hager, Georg; Zeiser, Thomas; Wellein, Gerhard (Proceedings of the 2008 IEEE International Parallel & Distributed Processing Symposium, 2008)
Processor and system architectures that feature multiple memory controllers are prone to show bottlenecks and erratic performance numbers on codes with regular access patterns. Although such effects are well known in the form of cache thrashing and aliasing conflicts, they become more severe when memory access is involved. Using the new Sun UltraSPARC T2 processor as a prototypical multi-core design, we analyze performance patterns in low-level and application benchmarks and show ways to circumvent bottlenecks by careful data layout and padding. ©2008 IEEE.
UnivIS-Import:2015-04-16:Pub.2008.zwiss.rrze.zensys.dataac_6

Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
Analytic, first-principles performance modeling of distributed-memory parallel codes is notoriously imprecise. Even for applications with extremely regular and homogeneous compute-communicate phases, simply adding communication time to computation time does often not yield a satisfactory prediction of parallel runtime due to deviations from the expected simple lockstep pattern caused by system noise, variations in communication time, and inherent load imbalance. In this paper, we highlight the specific cases of provoked and spontaneous desynchronization of memory-bound, bulk-synchronous pure MPI and hybrid MPI+OpenMP programs. Using simple microbenchmarks we observe that although desynchronization can introduce increased waiting time per process, it does not necessarily cause lower resource utilization but can lead to an increase in available bandwidth per core. In case of significant communication overhead, even natural noise can shove the system into a state of automatic overlap of communication and computation, improving the overall time to solution. The saturation point, i.e., the number of processes per memory domain required to achieve full memory bandwidth, is pivotal in the dynamics of this process and the emerging stable wave pattern. We also demonstrate how hybrid MPI-OpenMP programming can prevent desirable desynchronization by eliminating the bandwidth bottleneck among processes. A Chebyshev filter diagonalization application is used to demonstrate some of the observed effects in a realistic setting.
CRIS-Team Scopus Importer:2020-07-10

Direct numerical simulation of turbulent flow over dimples - Code optimization for NEC SX-8 plus flow results - Breuer, Michael; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard; Lammers, Peter (High Performance Computing in Science and Engineering '07: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2007, 2008)
The objective of the project DIMPLE is to study complex turbulent flows over a dimpled surface which is expected to lead to drag reduction compared to a smooth surface. In order to understand the physical mechanism of the flow over an arrangement of shallow dimples, CPU time-consuming direct numerical simulations were carried out using two different grid resolutions and three different configurations. As a first step within this project, the code LESOCC was evaluated and optimized for the new vector machine NEC SX-8. Classical vector systems still combine excellent performance with a well-established optimization approach. On the other hand, clusters based on commodity microprocessors offer comparable peak performance at much lower costs. In the context of the introduction of the NEC SX-8 vector computer series we compare single and parallel performance of a computational fluid dynamics application on the SX-8 and on the SGI Altix architecture demonstrating the potential of the SX-8 for teraflop computing in the area of turbulence research and flow control. Finally, results of the flow predictions over a dimpled surface are presented and compared with the reference case of plane channel flow with smooth walls.
UnivIS-Import:2015-04-20:Pub.2008.tech.ITC.stmmec.direct
Keywords: Dimples; Direct numerical simulation; Drag reduction; Flow control; High-performance computing; NEC SX-8; Sustained performance; Turbulence

Domain Decomposition and Locality Optimization for Large-Scale Lattice Boltzmann Simulations - Wittmann, Markus; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Computers & Fluids, 2012)
UnivIS-Import:2015-03-09:Pub.2012.tech.IMMD.phleir._domai

Efficient multicore-aware parallelization strategies for iterative stencil computations - Eitzinger, Jan; Wellein, Gerhard; Hager, Georg (Journal of Computational Science, 2011)
Stencil computations consume a major part of runtime in many scientific simulation codes. As prototypes for this class of algorithms we consider the iterative Jacobi and Gauss-Seidel smoothers and aim at highly efficient parallel implementations for cache-based multicore architectures. Temporal cache blocking is a known advanced optimization technique, which can reduce the pressure on the memory bus significantly. We apply and refine this optimization for a recently presented temporal blocking strategy designed to explicitly utilize multicore characteristics. Especially for the case of Gauss-Seidel smoothers we show that simultaneous multi-threading (SMT) can yield substantial performance improvements for our optimized algorithm on some architectures. © 2011 Elsevier B.V.
UnivIS-Import:2015-03-09:Pub.2011.tech.IMMD.phleir._effic
Keywords: Multicore; Simultaneous multi-threading; Spatial blocking; Stencil computations; Temporal blocking; Wavefront parallelization

Efficient optical simulation of nano structures in thin-film solar cells - Hornich, Julian; Hager, Georg; Pflaum, Christoph (2018)
Optical simulations are a crucial part in designing and understanding new solar cell devices to help with the transition to more environmentally friendly energy generation. We present a multicore wavefront diamond blocking scheme with multi-dimensional cache block sharing for our hybrid-parallel (MPI+OpenMP) optical solar cell simulation code utilizing the Time Harmonic Inverse Iteration Method (THIIM) to discretize Maxwell's equations. This approach allows to decouple the code from the main memory bandwidth bottleneck and achieves a single node speed-up of up to three compared to an optimal implementation with pure spatial blocking and a multi node speed-up of up to two even with large numbers of compute nodes.
Keywords: Simulation; Time Harmonic InverseIteration Method; Maxwell's Equations; High Performance Computing; Optical Simulation

Efficient temporal blocking for stencil computations by multicore-aware wavefront parallelization - Wellein, Gerhard; Hager, Georg; Zeiser, Thomas; Wittmann, Markus; Fehske, Holger (Proceedings of 2009 33rd Annual IEEE International Computer Software and Applications Conference, 2009)
We present a pipelined wavefront parallelization approach for stencil-based computations. Within a fixed spatial domain successive wavefronts are executed by threads scheduled to a multicore processor chip with a shared outer level cache. By re-using data from cache in the successive wavefronts this multicore-aware parallelization strategy employs temporal blocking in a simple and efficient way. We use the Jacobi algorithm in three dimensions as a prototype for stencil-based computations and prove the efficiency of our approach on the latest generations of Intel's x86 quad- and hexa-core processors. © 2009 IEEE.
UnivIS-Import:2015-04-16:Pub.2009.tech.IMMD.phleir._effic_8
Keywords: Multicore; Stencil computations; Temporal blocking; Wavefront parallelization

Energy-aware operation of HPC systems in Germany - Suarez, Estela; Bockelmann, Hendryk; Eicker, Norbert; Eitzinger, Jan; El Sayed, Salem; Fieseler, Thomas; Frank, Martin; Frech, Peter; Giesselmann, Pay; Hackenberg, Daniel; Hager, Georg; Herten, Andreas; et al. (Frontiers in High Performance Computing, 2025)
High Performance Computing (HPC) systems are among the most energy-intensive scientific facilities, with electric power consumption reaching and often exceeding 20 Megawatts per installation. Unlike other major scientific infrastructures such as particle accelerators or high-intensity light sources, which are few around the world, the number and size of supercomputers are continuously increasing. Even if every new system generation is more energy efficient than the previous one, the overall growth in size of the HPC infrastructure, driven by a rising demand for computational capacity across all scientific disciplines, and especially by Artificial Intelligence (AI) workloads, rapidly drives up the energy demand. This challenge is particularly significant for HPC centers in Germany, where high electricity costs, stringent national energy policies, and a strong commitment to environmental sustainability are key factors. This paper describes various state-of-the-art strategies and innovations employed to enhance the energy efficiency of HPC systems within the national context. Case studies from leading German HPC facilities illustrate the implementation of novel heterogeneous hardware architectures, advanced monitoring infrastructures, high-temperature cooling solutions, energy-aware scheduling, and dynamic power management, among other optimisations. By reviewing best practices and ongoing research, this paper aims to share valuable insight with the global HPC community, motivating the pursuit of more sustainable and energy-efficient HPC architectures and operation

ESSEX: Equipping Sparse Solvers for Exascale - Hager, Georg; Kreutzer, Moritz; Shahzad, Faisal; Wellein, Gerhard; et al. (Euro-Par 2014: Parallel Processing Workshops, 2014)
The ESSEX project investigates computational issues arising at exascale for large-scale sparse eigenvalue problems and develops programming concepts and numerical methods for their solution. The project pursues a coherent co-design of all software layers where a holistic performance engineering process guides code development across the classic boundaries of application, numerical method, and basic kernel library. Within ESSEX the numerical methods cover widely applicable solvers such as classic Krylov, Jacobi-Davidson, or the recent FEAST methods, as well as domain-specific iterative schemes relevant for the ESSEX quantum physics application. This report introduces the project structure and presents selected results which demonstrate the potential impact of ESSEX for efficient sparse solvers on highly scalable heterogeneous supercomputers.
UnivIS-Import:2015-04-20:Pub.2014.tech.IMMD.phleir.essexe

Evaluation of the Coarray Fortran Programming Model on the Example of a Lattice Boltzmann Code - Sembritzki, Klaus; Hager, Georg; Krammer, Bettina; Eitzinger, Jan; Wellein, Gerhard (PGAS12, 2012)
UnivIS-Import:2015-04-16:Pub.2012.tech.IMMD.phleir._evalu

Execution-Cache-Memory modeling and performance tuning of sparse matrix-vector multiplication and Lattice quantum chromodynamics on A64FX - Alappat, Christie; Meyer, Nils; Laukemann, Jan; Gruber, Thomas; Hager, Georg; Wellein, Gerhard; Wettig, Tilo (Concurrency and Computation-Practice & Experience, 2021)
Keywords: A64FX; ECM model; Lattice quantum chromodynamics; sparse matrix-vector multiplication

Exploring performance and power properties of modern multi-core chips via simple machine models - Hager, Georg; Eitzinger, Jan; Habich, Johannes; Wellein, Gerhard (Concurrency and Computation-Practice & Experience, 2016)
Modern multi-core chips show complex behavior with respect to performance and power. Starting with the Intel Sandy Bridge processor, it has become possible to directly measure the power dissipation of a CPU chip and correlate this data with the performance properties of the running code. Going beyond a simple bottleneck analysis, we employ the recently published Execution-Cache-Memory (ECM) model to describe the single-core and multi-core performance of streaming kernels. The model refines the well-known roofline model, because it can predict the scaling and the saturation behavior of bandwidth-limited loop kernels on a multi-core chip. The saturation point is especially relevant for considerations of energy consumption. From power dissipation measurements of benchmark programs with vastly different requirements to the hardware, we derive a simple, phenomenological power model for the Sandy Bridge processor. Together with the ECM model, we are able to explain many peculiarities in the performance and power behavior of multi-core processors and derive guidelines for energy-efficient execution of parallel programs. Finally, we show that the ECM and power models can be successfully used to describe the scaling and power behavior of a lattice Boltzmann flow solver code. Copyright (c) 2013 John Wiley & Sons, Ltd.
Keywords: multi-core; power modeling; performance modeling; ECM model

Exploring performance and power properties of modern multicore chips via simple machine models - Hager, Georg; Eitzinger, Jan; Wellein, Gerhard; Habich, Johannes (Concurrency and Computation-Practice & Experience, 2012)
UnivIS-Import:2015-03-09:Pub.2012.tech.IMMD.phleir._explo

Exploring Techniques for the Analysis of Spontaneous Asynchronicity in MPI-Parallel Applications - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard; Markidis, Stefano (Lecture Notes in Computer Science, 2023)
This paper studies the utility of using data analytics and machine learning techniques for identifying, classifying, and characterizing the dynamics of large-scale parallel (MPI) programs. To this end, we run microbenchmarks and realistic proxy applications with the regular compute-communicate structure on two different supercomputing platforms and choose the per-process performance and MPI time per time step as relevant observables. Using principal component analysis, clustering techniques, correlation functions, and a new “phase space plot,” we show how desynchronization patterns (or lack thereof) can be readily identified from a data set that is much smaller than a full MPI trace. Our methods also lead the way towards a more general classification of parallel program dynamics.<br
Keywords: Parallel distributed computing; Data analytic techniques; Machine learning techniques; Asynchronous MPI execution; Scalability and bottleneck

Expression Templates Revisited: A Performance Analysis of the Current ET Methodology - Iglberger, Klaus; Hager, Georg; Eitzinger, Jan; Rüde, Ulrich (SIAM Journal on Scientific Computing, 2011)
In the last decade, Expression Templates (ET) have gained a reputation as an efficient performance optimization tool for C++ codes. This reputation builds on several ET-based linear algebra frameworks focused on combining both elegant and high-performance C++ code. However, on closer examination the assumption that ETs are a performance optimization technique cannot be maintained. In this paper we demonstrate and explain the inability of current ET-based frameworks to deliver high performance for dense and sparse linear algebra operations, and introduce a new ET implementation that truly allows the combination of high performance code with the elegance and maintainability of a domain-specific language.
UnivIS-Import:2015-03-09:Pub.2011.tech.IMMD.lsinfs.expres

Fission of super-heavy nuclei explored with skyrme forces - Schindzielorz, Nils; Erler, J.; Klüpfel, P.; Reinhard, Paul-Gerhard; Hager, Georg (International Journal of Modern Physics E-Nuclear Physics, 2009)
We present a large scale survey of life-times for spontaneous fission in the regime of super-heavy elements (SHE), i.e. nuclei with Z=104-122. This is done on the basis of the Skyrme-Hartree-Fock model. The axially symmetric fission path is computed using a quadrupole constraint. Self-consistent cranking is used for the collective masses and associated quantum corrections. The actual tunneling probability is estimated by the WKB approximation. Three typical Skyrme forces are used to explore the sensitivity of the results. Benchmarks in the regime Z=104-108 show an acceptable agreement. The general systematics reflects nicely the islands of shell stabilization and the crossover from α-decay to fission for the decay chains from the region of Z/N=118/176. © 2009 World Scientific Publishing Company.
CRIS-Team Scopus Importer:2023-07-24

GHOST: Building Blocks for High Performance Sparse Linear Algebra on Heterogeneous Systems - Kreutzer, Moritz; Thies, Jonas; Röhrig-Zöllner, Melven; Pieper, Andreas; Shahzad, Faisal; Galgon, Martin; Basermann, Achim; Fehske, Holger; Hager, Georg; Wellein, Gerhard (International Journal of Parallel Programming, 2016)
While many of the architectural details of future exascale-class high performance computer systems are still a matter of intense research, there appears to be a general consensus that they will be strongly heterogeneous, featuring “standard” as well as “accelerated” resources. Today, such resources are available as multicore processors, graphics processing units (GPUs), and other accelerators such as the Intel Xeon Phi. Any software infrastructure that claims usefulness for such environments must be able to meet their inherent challenges: massive multi-level parallelism, topology, asynchronicity, and abstraction. The “General, Hybrid, and Optimized Sparse Toolkit” (GHOST) is a collection of building blocks that targets algorithms dealing with sparse matrix representations on current and future large-scale systems. It implements the “MPI+X” paradigm, has a pure C interface, and provides hybrid-parallel numerical kernels, intelligent resource management, and truly heterogeneous parallelism for multicore CPUs, Nvidia GPUs, and the Intel Xeon Phi. We describe the details of its design with respect to the challenges posed by modern heterogeneous supercomputers and recent algorithmic developments. Implementation details which are indispensable for achieving high efficiency are pointed out and their necessity is justified by performance measurements or predictions based on performance models. We also provide instructions on how to make use of GHOST in existing software packages, together with a case study which demonstrates the applicability and performance of GHOST as a component within a larger software stack. The library code and several applications are available as open sourc
Keywords: Data parallelism; Heterogeneous computing; Large scale computing; Software library; Sparse linear algebra; Task parallelism

Have the Vectors the Continuing Ability to Parry the Attack of the Killer Micros? - Lammers, Peter; Wellein, Gerhard; Zeiser, Thomas; Hager, Georg; Breuer, Michael (High Performance Computing on Vector Systems: Proceedings of the High Performance Computing Center Stuttgart, March 2005, 2006)
UnivIS-Import:2015-04-20:Pub.2006.zwiss.rrze.zensys.haveth

High-performance implementation of Chebyshev filter diagonalization for interior eigenvalue computations - Wellein, Gerhard; Alvermann, Andreas; Fehske, Holger; Hager, Georg; Kreutzer, Moritz; Lang, Bruno; Pieper, Andreas; Galgon, Martin (Journal of Computational Physics, 2016)
We study Chebyshev filter diagonalization as a tool for the computation of many interior eigenvalues of very large sparse symmetric matrices. In this technique the subspace projection onto the target space of wanted eigenvectors is approximated with filter polynomials obtained from Chebyshev expansions of window functions. After the discussion of the conceptual foundations of Chebyshev filter diagonalization we analyze the impact of the choice of the damping kernel, search space size, and filter polynomial degree on the computational accuracy and effort, before we describe the necessary steps towards a parallel high-performance implementation. Because Chebyshev filter diagonalization avoids the need for matrix inversion it can deal with matrices and problem sizes that are presently not accessible with rational function methods based on direct or iterative linear solvers. To demonstrate the potential of Chebyshev filter diagonalization for large-scale problems of this kind we include as an example the computation of the 10(2) innermost eigenpairs of a topological insulator matrix with dimension 10(9) derived from quantum physics applications. (C) 2016 Elsevier Inc. All rights reserved.
Keywords: Interior eigenvalues; Chebyshev filter polynomials; Performance engineering; Quantum physics; Topological materials

High performance smart expression template math libraries - Iglberger, Klaus; Hager, Georg; Eitzinger, Jan; Rüde, Ulrich (High Performance Computing and Simulation (HPCS) 2012, International Conference on, 2012)
Performance is of utmost importance for linear algebra libraries since they usually are the core of numerical and simulation packages and use most of the available compute time and resources. However, especially in large scale simulation frameworks the readability and ease of use of mathematical expressions is essential for a continuous maintenance, modification, and extension of the software framework. Based on these requirements, in the last decade C++ Expression Templates have gained a reputation as a suitable means to combine an elegant, domain-specific, and intuitive user interface with "HPC-grade" performance. Unfortunately, many of the available ET-based frameworks fall short of the expectation to deliver high performance, adding to the general mistrust towards C++ math libraries. In this paper we present performance results for Smart Expression Template libraries, demonstrating that by proper combination of high-level C++ code and low-level compute kernels both requirements, an elegant interface and high performance, can be achieved. © 2012 IEEE.
UnivIS-Import:2015-04-16:Pub.2012.tech.IMMD.lsinfs.highpe

Hybrid-parallel sparse matrix-vector multiplication with explicit communication overlap on current multicore-based systems. - Hager, Georg; Wellein, Gerhard; Schubert, Gerald; Fehske, Holger (Parallel Processing Letters, 2011)
We evaluate optimized parallel sparse matrix-vector operations for several representative application areas on widespread multicore-based cluster configurations. First the single-socket baseline performance is analyzed and modeled with respect to basic architectural properties of standard multicore chips. Beyond the single node, the performance of parallel sparse matrix-vector operations is often limited by communication overhead. Starting from the observation that nonblocking MPI is not able to hide communication cost using standard MPI implementations, we demonstrate that explicit overlap of communication and computation can be achieved by using a dedicated communication thread, which may run on a virtual core. Moreover we identify performance benefits of hybrid MPI/OpenMP programming due to improved load balancing even without explicit communication overlap. We compare performance results for pure MPI, the widely used "vector-like" hybrid programming strategies, and explicit overlap on a modern multicore-based cluster and a Cray XE6 system. © 2011 World Scientific Publishing Company.
UnivIS-Import:2015-03-09:Pub.2011.tech.IMMD.phleir.hybrid
Keywords: hybrid parallelization; multicore; sparse matrix-vector multiply

Improved coefficients for polynomial filtering in ESSEX - Galgon, Martin; Krämer, Lukas; Lang, Bruno; Alvermann, Andreas; Fehske, Holger; Pieper, Andreas; Hager, Georg; Kreutzer, Moritz; Shahzad, Faisal; Wellein, Gerhard; Basermann, Achim; Röhrig-Zöllner, Melven; et al. (1st InternationalWorkshop on Eigenvalue Problems: Algorithms, Software and Applications in Petascale Computing, EPASA 2015, 2017)

Increasing the performance of the Jacobi-Davidson method by blocking - Röhrig-Zöllner, Melven; Thies, Jonas; Kreutzer, Moritz; Alvermann, Andreas; Pieper, Andreas; Basermann, Achim; Hager, Georg; Wellein, Gerhard; Fehske, Holger (SIAM Journal on Scientific Computing, 2015)
Block variants of the Jacobi-Davidson method for computing a few eigenpairs of a large sparse matrix are known to improve the robustness of the standard algorithm when it comes to computing multiple or clustered eigenvalues. In practice, however, they are typically avoided because the total number of matrix-vector operations increases. In this paper we present the implementation of a block Jacobi-Davidson solver. By detailed performance engineering and numerical experiments we demonstrate that the increase in operations is typically more than compensated by performance gains through better cache usage on modern CPUs, resulting in a method that is both more efficient and robust than its single vector counterpart. The steps to be taken to achieve a block speedup involve both kernel optimizations for sparse matrix and block vector operations, and algorithmic choices to allow using blocked operations in most parts of the computation. We discuss the aspect of avoiding synchronization in the algorithm and show by numerical experiments with our hybrid parallel implementation that a significant speedup through blocking can be achieved for a variety of matrices on up to 5 120 CPU cores as long as at least about 20 eigenpairs are sought.
UnivIS-Import:2016-06-01:Pub.2015.tech.IMMD.phleir.increa
Keywords: Block methods; High performance computing; Hybrid parallel implementation; Jacobi-Davidson; Multicore processors; Performance engineering; Sparse eigenvalue problems

Introducing a parallel cache oblivious blocking approach for the lattice Boltzmann method - Zeiser, Thomas; Wellein, Gerhard; Iglberger, Klaus; Rüde, Ulrich; Hager, Georg; Nitsure, Aditya (Progress in Computational Fluid Dynamics, 2008)
In this report we propose a parallel cache oblivious spatial and temporal blocking algorithm for the lattice Boltzmann method in three spatial dimensions. The algorithm has originally been proposed by Frigo et al. (1999) and divides the space-time domain of stencil-based methods in an optimal way, independently of any external parameters, e.g., cache size. In view of the increasing gap between processor speed and memory performance this approach offers a promising path to increase cache utilisation. We find that even a straightforward cache oblivious implementation can reduce memory traffic at least by a factor of two if compared to a highly optimised standard kernel and improves scalability for shared memory parallelisation. Due to the recursive structure of the algorithm we use an unconventional parallelisation scheme based on task queuing. Copyright © 2008, Inderscience Publishers.
UnivIS-Import:2015-03-09:Pub.2008.tech.IMMD.lsinfs._intro
Keywords: Cache oblivious; Cache optimisation; Lattice Boltzmann; Multi core; Shared memory parallelisation; Task queuing

Introducing a Performance Model for Bandwidth-Limited Loop Kernels - Eitzinger, Jan; Hager, Georg (Parallel Processing and Applied Mathematics, 2010)
We present a diagnostic performance model for bandwidth-limited loop kernels which is founded on the analysis of modern cache based microarchitectures. This model allows an accurate performance prediction and evaluation for existing instruction codes. It provides an in-depth understanding of how performance for different memory hierarchy levels is made up. The performance of raw memory load, store and copy operations and a stream vector triad are analyzed and benchmarked on three modern x86-type quad-core architectures in order to demonstrate the capabilities of the model. © 2010 Springer-Verlag Berlin Heidelberg.
UnivIS-Import:2015-04-16:Pub.2010.tech.IMMD.phleir._intro

Kerncraft: A Tool for Analytic Performance Modeling of Loop Kernels - Hammer, Julian; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Tools for High Performance Computing 2016, 2017)
Achieving optimal program performance requires deep insight into the interaction between hardware and software. For software developers without an in-depth background in computer architecture, understanding and fully utilizing modern architectures is close to impossible. Analytic loop performance modeling is a useful way to understand the relevant bottlenecks of code execution based on simple machine models. The Roofline Model and the Execution-Cache-Memory (ECM) model are proven approaches to performance modeling of loop nests. In comparison to the Roofline model, the ECM model can also describes the single-core performance and saturation behavior on a multicore chip.We give an introduction to the Roofline and ECM models, and to stencil performance modeling using layer conditions (LC). We then present Kerncraft, a tool that can automatically construct Roofline and ECM models for loop nests by performing the required code, data transfer, and LC analysis. The layer condition analysis allows to predict optimal spatial blocking factors for loop nests. Together with the models it enables an ab-initio estimate of the potential benefits of loop blocking optimizations and of useful block sizes. In cases where LC analysis is not easily possible, Kerncraft supports a cache simulator as a fallback option. Using a 25-point long-range stencil we demonstrate the usefulness and predictive power of the Kerncraft tool.

Level-based Blocking for Sparse Matrices: Sparse Matrix-Power-Vector Multiplication - Alappat, Christie; Hager, Georg; Schenk, Olaf; Wellein, Gerhard (IEEE Transactions on Parallel and Distributed Systems, 2022)
The multiplication of a sparse matrix with a dense vector (SpMV) is a key component in many numerical schemes and its performance is known to be severely limited by main memory access. Several numerical schemes require the multiplication of a sparse matrix polynomial with a dense vector which is typically implemented as a sequence of SpMVs. This results in low performance and ignores the potential to increase the arithmetic intensity by reusing the matrix data from cache. In this work we use the recursive algebraic coloring engine (RACE) to enable blocking of sparse matrix data across the polynomial computations. In the graph representing the sparse matrix we form levels using a breadth-first search. Locality relations of these levels are then used to improve spatial and temporal locality when accessing the matrix data and to implement an efficient multithreaded parallelization. Our approach is independent of the matrix structure and avoids shortcomings of existing “blocking” strategies in terms of hardware efficiency and parallelization overhead. We quantify the quality of our implementation using performance modelling and demonstrate speedups of up to 3× and 5× compared to an optimal SpMV-based baseline on a single multicore chip of recent Intel and AMD architectures. Various numerical schemes like s-step Krylov solvers, polynomial preconditioners and power clustering algorithms will benefit from our development.
CRIS-Team Scopus Importer:2022-12-30
Keywords: Algorithm design and analysis; Bandwidth; Computer architecture; Computer architecture; Graph algorithms; Hardware; Kernel; Kernel optimization; Memory hierarchies; Multicore processing; Optimization; Performance evaluation; Sparse matrices; Sparse matrices

Leveraging shared caches for parallel temporal blocking of stencil codes on multicore processors and clusters - Wittmann, Markus; Hager, Georg; Eitzinger, Jan; Wellein, Gerhard (Parallel Processing Letters, 2010)
Bandwidth-starved multicore chips have become ubiquitous. It is well known that the performance of stencil codes can be improved by temporal blocking, lessening the pressure on the memory interface. We introduce a new pipelined approach that makes explicit use of shared caches in multicore environments and minimizes synchronization and boundary overhead. Benchmark results are presented for three current x86-based microprocessors, showing clearly that our optimization works best on designs with high-speed shared caches and low memory bandwidth per core. We furthermore demonstrate that simple bandwidth-based performance models are inaccurate for this kind of algorithm and employ a more elaborate, synthetic modeling procedure. Finally we show that temporal blocking can be employed successfully in a hybrid shared/distributed-memory environment, albeit with limited benefit at strong scaling. © 2010 World Scientific Publishing Company.
UnivIS-Import:2015-03-09:Pub.2010.tech.IMMD.phleir._lever
Keywords: multi-halo exchange; multicore; shared caches; stencil algorithm; temporal blocking

LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments - Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Proceedings of PSTI2010, the First International Workshop on Parallel Software Tools and Tool Infrastructures, 2010)
Exploiting the performance of today's processors requires intimate knowledge of the microarchitecture as well as an awareness of the ever-growing complexity in thread and cache topology. LIKWID is a set of command-line utilities that addresses four key problems: Probing the thread and cache topology of a shared-memory node, enforcing threadcore affinity on a program, measuring performance counter metrics, and toggling hardware prefetchers. An API for using the performance counting features from user code is also included. We clearly state the differences to the widely used PAPI interface. To demonstrate the capabilities of the tool set we show the influence of thread pinning on performance using the well-known OpenMP STREAM triad benchmark, and use the affinity and hardware counter tools to study the performance of a stencil code specifically optimized to utilize shared caches on multicore chips. © 2010 IEEE.
UnivIS-Import:2015-04-16:Pub.2010.tech.IMMD.phleir._likwi_6

LIKWID monitoring stack: A flexible framework enabling job specific performance monitoring for the masses - Röhl, Thomas; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (2017 IEEE International Conference on Cluster Computing, CLUSTER 2017, 2017)
Keywords: Hardware performance; High performance computing; Job specific performance monitoring; Likwid monitoring stack; Medium sized commodity cluster; Monitoring infrastructure; Performance monitoring; System wide performance monitoring

LIKWID performance tools - Eitzinger, Jan; Hager, Georg; Wellein, Gerhard; Meier, Michael (2010)
UnivIS-Import:2015-03-09:Pub.2010.tech.IMMD.phleir._likwi

Making applications faster by asynchronous execution: Slowing down processes or relaxing MPI collectives - Afzal, Ayesha; Hager, Georg; Markidis, Stefano; Wellein, Gerhard (Future Generation Computer Systems-The International Journal of Grid Computing Theory Methods and Applications, 2023)
Comprehending the performance bottlenecks at the core of the intricate hardware-software interactions exhibited by highly parallel programs on HPC clusters is crucial. This paper sheds light on the issue of automatically asynchronous MPI communication in memory-bound parallel programs on multicore clusters and how it can be facilitated. For instance, slowing down MPI processes by deliberate injection of delays can improve performance if certain conditions are met. This leads to the counter-intuitive conclusion that noise, independent of its source, is not always detrimental but can be leveraged for performance improvements. We employ phase-space graphs as a new tool to visualize parallel program dynamics. They are useful in spotting certain patterns in parallel execution that will easily go unnoticed with traditional tracing tools. We investigate five different microbenchmarks and applications on different supercomputer platforms: an MPI-augmented STREAM Triad, two implementations of Lattice-Boltzmann fluid solvers (D3Q19 and SPEChpc D2Q37), the LULESH and HPCG proxy applications.<br
Keywords: parallel distributed computing; data analytic techniques; MPI collectives; asynchronous MPI execution; resource scalability and bottleneck.

MD-Bench: A performance-focused prototyping harness for state-of-the-art short-range molecular dynamics algorithms - Ravedutti Lucio Machado, Rafael; Eitzinger, Jan; Laukemann, Jan; Hager, Georg; Köstler, Harald; Wellein, Gerhard (Future Generation Computer Systems-The International Journal of Grid Computing Theory Methods and Applications, 2023)
Molecular dynamics (MD) simulations provide considerable benefits for the investigation and experimentation of systems at atomic level. Their usage is widespread into several research fields, but their system size and timescale are crucially limited by the available computing power. Performance engineering of MD kernels is therefore critical to understand their bottlenecks and investigate possible improvements. For that reason, we developed MD-Bench, a performance-focused prototyping harness for short-range MD kernels that implements state-of-the-art algorithms from multiple production applications such as LAMMPS and GROMACS. The MD-Bench source code is simple, understandable, and extensible, and therefore well suited for benchmarking, teaching, and researching MD algorithms. In this paper we introduce MD-Bench, describe its design, structure, and implemented algorithms. Finally, we show five use-cases of MD-Bench and describe how these are useful to gain a deeper understanding of the performance of MD kernels.<br

MD-Bench: A performance-focused prototyping harness for state-of-the-art short-range molecular dynamics algorithms - Ravedutti Lucio Machado, Rafael; Eitzinger, Jan; Laukemann, Jan; Hager, Georg; Köstler, Harald; Wellein, Gerhard (Future Generation Computer Systems-The International Journal of Grid Computing Theory Methods and Applications, 2023)
Molecular dynamics (MD) simulations provide considerable benefits for the investigation and experimentation of systems at atomic level. Their usage is widespread into several research fields, but their system size and timescale are crucially limited by the available computing power. Performance engineering of MD kernels is therefore critical to understand their bottlenecks and investigate possible improvements. For that reason, we developed MD-Bench, a performance-focused prototyping harness for short-range MD kernels that implements state-of-the-art algorithms from multiple production applications such as LAMMPS and GROMACS. The MD-Bench source code is simple, understandable, and extensible, and therefore well suited for benchmarking, teaching, and researching MD algorithms. In this paper we introduce MD-Bench, describe its design, structure, and implemented algorithms. Finally, we show five use-cases of MD-Bench and describe how these are useful to gain a deeper understanding of the performance of MD kernels.<br

Microarchitectural comparison and in-core modeling of state-of-the-art CPUs: Grace, Sapphire Rapids, and Genoa - Laukemann, Jan; Hager, Georg; Wellein, Gerhard (SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2024)
With Nvidia’s release of the Grace Superchip, all three big semiconductor companies in HPC (AMD, Intel, Nvidia) are currently competing in the race for the best CPU. In this work we analyze the performance of these state-of-the-art CPUs and create an accurate in-core performance model for their microarchitectures Zen 4, Golden Cove, and Neoverse V2, extending the Open Source Architecture Code Analyzer (OSACA) tool and comparing it with LLVM-MCA. Starting from the peculiarities and up- and downsides of a single core, we extend our comparison by a variety of microbenchmarks and the capabilities of a full node. The "write-allocate (WA) evasion" feature, which can automatically reduce the memory traffic caused by write misses, receives special attention; we show that the Grace Superchip has a next-to-optimal implementation of WA evasion, and that the only way to avoid write allocates on Zen 4 is the explicit use of non-temporal store
Keywords: Semiconductor device modeling; Analytical models; Solid modeling; Microarchitecture; Codes; Accuracy; Runtime; Data models; Servers; Standards

Model-guided Performance Analysis of the Sparse Matrix-Matrix Multiplication - Scharpff, Tobias; Iglberger, Klaus; Hager, Georg; Rüde, Ulrich (HPCS2013 Conference Proceedings (Preprint bei arXiv), 2013)
Achieving high efficiency with numerical kernels for sparse matrices is of utmost importance, since they are part of many simulation codes and tend to use most of the available compute time and resources. In addition, especially in large scale simulation frameworks the readability and ease of use of mathematical expressions are essential components for the continuous maintenance, modification, and extension of software. In this context, the sparse matrix-matrix multiplication is of special interest. In this paper we thoroughly analyze the single-core performance of sparse matrix-matrix multiplication kernels in the Blaze Smart Expression Template (SET) framework. We develop simple models for estimating the achievable maximum performance, and use them to assess the efficiency of our implementations. Additionally, we compare these kernels with several commonly used SET-based C++ libraries, which, just as Blaze, aim at combining the requirements of high performance with an elegant user interface. For the different sparse matrix structures considered here, we show that our implementations are competitive or faster than those of the other SET libraries for most problem sizes on a current Intel multicore processor. © 2013 IEEE.
UnivIS-Import:2015-04-16:Pub.2013.tech.IMMD.lsinfs.modelg

Modeling and analyzing performance for highly optimized propagation steps of the lattice Boltzmann method on sparse lattices - Wittmann, Markus; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (2014)
UnivIS-Import:2015-03-09:Pub.2014.tech.IMMD.phleir.modeli

MPC and CoArray Fortran: Alternatives to classic MPI implementations on the examples of scalable lattice boltzmann flow solvers - Wittmann, Markus; Hager, Georg; Wellein, Gerhard; Zeiser, Thomas; Krammer, B. (15th Results and Review Workshop on High Performance Computing in Science and Engineering, HLRS 2012, 2013)
In recent years, more and more parallel programming concepts have emerged as alternatives or improvements to the well established MPI concept. Arguments for all the new parallel languages or alternative communication frameworks are typically the increasing number of cores in modern systems and the hierarchical memory structure in clusters of multi-socket multi-core compute nodes. © Springer-Verlag Berlin Heidelberg 201

Multicore-optimized wavefront diamond blocking for optimizing stencil updates - Malas, Tareq; Hager, Georg; Ltaief, Hatem; Stengel, Holger; Wellein, Gerhard; Keyes, David (SIAM Journal on Scientific Computing, 2015)
Keywords: Diamond tiling; Multicore; Energy-efficient algorithms; Temporal blocking; Stencil computations; Wavefront parallelization

Multicore Performance Engineering of Sparse Triangular Solves Using a Modified Roofline Model - Wittmann, Markus; Hager, Georg; Janalik, Radim; Lanser, Martin; Klawonn, Axel; Rheinbach, Oliver; Schenk, Olaf; Wellein, Gerhard (2018 30TH INTERNATIONAL SYMPOSIUM ON COMPUTER ARCHITECTURE AND HIGH PERFORMANCE COMPUTING (SBAC-PAD 2018), 2018)
CRIS-Team WoS Importer:2019-04-12

On the Accuracy and Usefulness of Analytic Energy Models for Contemporary Multicore Processors - Hofmann, Johannes; Hager, Georg; Fey, Dietmar (High Performance Computing: 33rd International Conference, ISC High Performance 2018, 2018)
This paper presents refinements to the execution-cache-memory performance model and a previously published power model for multicore processors. The combination of both enables a very accurate prediction of performance and energy consumption of contemporary multicore processors as a function of relevant parameters such as number of active cores as well as core and Uncore frequencies. Model validation is performed on Intel Sandy Bridge-EP, Broadwell-EP, and AMD Epyc processors. Production-related variations in chip quality are demonstrated through a statistical analysis of the fit parameters obtained on one hundred Broadwell-EP CPUs of the same model. Insights from the models are used to explain the performance- and energy-related behavior of the processors for scalable as well as saturating (i.e., memory-bound) codes. In the process we demonstrate the models’ capability to identify optimal operating points with respect to highest performance, lowest energy-to-solution, and lowest energy-delay product and identify a set of best practices for energy-efficient execution.
UnivIS-Import:2018-09-06:Pub.2018.tech.IMMD.IMMD3.onthea
Keywords: Performance modeling; Power modeling; Energy modeling

On the single processor performance of simple lattice Boltzmann kernels - Wellein, Gerhard; Zeiser, Thomas; Hager, Georg; Donath, Stefan (Computers & Fluids, 2006)
This report presents a comprehensive survey of the effect of different data layouts on the single processor performance characteristics for the lattice Boltzmann method both for commodity "off-the-shelf" (COTS) architectures and tailored HPC systems, such as vector computers. We cover modern 64-bit processors ranging from IA32 compatible (Intel Xeon/Nocona, AMD Opteron), superscalar RISC (IBM Power4), IA64 (Intel Itanium 2) to classical vector (NEC SX6+) and novel vector (Cray X1) architectures. Combining different data layouts with architecture dependent optimization strategies we demonstrate that the optimal implementation strongly depends on the architecture used. In particular, the correct choice of the data layout could supersede complex cache-blocking techniques in our kernels. Furthermore our results demonstrate that vector systems can outperform COTS architectures by one order of magnitude. © 2005 Elsevier Ltd. All rights reserved.
UnivIS-Import:2015-03-09:Pub.2006.tech.IMMD.lsinfs._onthe

Opening the Black Box: Performance Estimation during Code Generation for GPUs - Ernst, Dominik; Hager, Georg; Holzer, Markus; Knorr, Matthias; Wellein, Gerhard (IEEE 33rd International Symposium on Computer Architecture and High Performance Computing, 2021)
Automatic code generation is frequently used to create implementations of algorithms specifically tuned to particular hardware and application parameters. The code generation process involves the selection of adequate code transformations, tuning parameters, and parallelization strategies. To cover the huge search space, code generation frameworks may apply time-intensive autotuning, exploit scenario-specific performance models, or treat performance as an intangible black box that must be described via machine learning. This paper addresses the selection problem by identifying the relevant performance-defining mechanisms through a performance model coupled with an analytic hardware metric estimator. This enables a quick exploration of large configuration spaces to identify highly efficient candidates with high accuracy. Our current approach targets memory-intensive GPGPU applications and focuses on the correct modeling of data transfer volumes to all levels of the memory hierarchy. We show how our method can be coupled to the pystencils stencil code generator, which is used to generate kernels for a range four 3D25pt stencil and a complex two phase fluid solver based on the Lattice Boltzmann Method. For both, it delivers a ranking that can be used to select the best performing candidate. The method is not limited to stencil kernels, but can be integrated into any code generator that can generate the required address expressions. </block

Optimization of Cache Oblivious Lattice Boltzmann Method in 2D and 3D - Iglberger, Klaus; Rüde, Ulrich; Feichtinger, Christian; Wellein, Gerhard; Hager, Georg; Nitsure, Aditya (Simulationstechnique - 19th Symposium in Hannover, September 2006, 2006)
UnivIS-Import:2015-04-16:Pub.2006.tech.IMMD.lsinfs.optimi_9

Optimizing performance of the lattice Boltzmann method for complex structures on cache-based architectures - Donath, Stefan; Zeiser, Thomas; Hager, Georg; Habich, Johannes; Wellein, Gerhard (Frontiers in Simulation: Simulationstechnique, 18th Symposium in Erlangen, September 2005 (ASIM), 2005)
UnivIS-Import:2015-04-16:Pub.2005.tech.IMMD.lsinfs._optim

Optimizing performance on modern HPC systems: learning from simple kernel benchmarks - Hager, Georg; Zeiser, Thomas; Eitzinger, Jan; Wellein, Gerhard (Computational Science and High Performance Computing II, 2006)
We discuss basic optimization and parallelization strategies for current cache-based microprocessors (Intel Itanium2, Intel Netburst and AMD64 variants) in single-CPU and shared memory environments. Using selected kernel benchmarks representing data intensive applications we focus on the effective bandwidths attainable, which is still suboptimal using current compilers. We stress the need for a subtle OpenMP implementation even for simple benchmark programs, to exploit the high aggregate memory bandwidth available nowadays on ccNUMA systems. If the quality of main memory access is the measure, classical vector systems such as the NEC SX6+ are still a class of their own and are able to sustain the performance level of in-cache operations of modern microprocessors even with arbitrarily large data sets. © 2006 Springer-Verlag Berlin Heidelberg.
UnivIS-Import:2015-04-16:Pub.2006.zwiss.rrze.zensys.optimi

Orthogonal Layers of Parallelism in Large-Scale Eigenvalue Computations - Alvermann, Andreas; Hager, Georg; Fehske, Holger (ACM Transactions on Parallel Computing, 2023)
We address the communication overhead of distributed sparse matrix-(multiple)-vector multiplication in the context of large-scale eigensolvers, using filter diagonalization as an example. The basis of our study is a performance model, which includes a communication metric that is computed directly from the matrix sparsity pattern without running any code. The performance model quantifies to which extent scalability and parallel efficiency are lost due to communication overhead.To restore scalability, we identify two orthogonal layers of parallelism in the filter diagonalization technique. In the horizontal layer the rows of the sparse matrix are distributed across individual processes. In the vertical layer bundles of multiple vectors are distributed across separate process groups. An analysis in terms of the communication metric predicts that scalability can be restored if, and only if, one implements the two orthogonal layers of parallelism via different distributed vector layouts.Our theoretical analysis is corroborated by benchmarks for application matrices from quantum and solid state physics, road networks, and nonlinear programming. We finally demonstrate the benefits of using orthogonal layers of parallelism with two exemplary application cases - an exciton and a strongly correlated electron system - which incur either small or large communication overhead.
CRIS-Team Scopus Importer:2023-10-20
Keywords: communication-avoiding algorithms; Distributed computing; sparse matrix-vector multiplication

Overhead Analysis of Performance Counter Measurements - Wellein, Gerhard; Eitzinger, Jan; Hager, Georg; Röhl, Thomas (43rd International Conference on Parallel Processing Workshops, ICPPW 2014, 2015)
Keywords: Hardware Performance Counters; Profiling; Overhead; X86; Tools

Parallel sparse matrix-vector multiplication as a test case for hybrid MPI OpenMP programming - Schubert, Gerald; Hager, Georg; Fehske, Holger; Wellein, Gerhard (25th IEEE International Parallel and Distributed Processing Symposium, Workshops and Phd Forum, IPDPSW 2011, 2011)
Keywords: Hybrid parallelization; Sparse matrix-vector multiply; Multicore

Performance analysis and optimization strategies for a D3Q19 lattice Boltzmann kernel on nVIDIA GPUs using CUDA - Habich, Johannes; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Advances in Engineering Software, 2011)
This paper presents implementation strategies and optimization approaches for a D3Q19 lattice Boltzmann flow solver on nVIDIA graphics processing units (GPUs). Using the STREAM benchmarks we demonstrate the GPU parallelization approach and obtain an upper limit for the flow solver performance. We discuss the GPU-specific implementation of the solver with a focus on memory alignment and register shortage. The optimized code is up to an order of magnitude faster than standard two-socket x86 servers with AMD Barcelona or Intel Nehalem CPUs. We further analyze data transfer rates for the PCI-express bus to evaluate the potential benefits of multi-GPU parallelism in a cluster environment. © 2010 Civil-Comp Ltd and Elsevier Ltd. All rights reserved.
UnivIS-Import:2015-04-16:Pub.2011.tech.IMMD.phleir._perfo
Keywords: CFD; GPU; Lattice Boltzmann; HPC; STREAM benchmarks; CUDA

Performance analysis of the Kahan-enhanced scalar product on current multi-core and many-core processors - Hofmann, Johannes; Fey, Dietmar; Riedmann, Michael; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Concurrency and Computation-Practice & Experience, 2017)
We investigate the performance characteristics of a numerically enhanced scalar product (dot) kernel loop that uses the Kahan algorithm to compensate for numerical errors, and describe efficient single instruction multiple data-vectorized implementations on recent multi-core and many-core processors. Using low-level instruction analysis and the execution-cache-memory performance model, we pinpoint the relevant performance bottlenecks for single-core and thread-parallel execution and predict performance and saturation behavior. We show that the Kahan-enhanced scalar product comes at almost no additional cost compared with the naive (non-Kahan) scalar product if appropriate low-level optimizations, notably single instruction multiple data vectorization and unrolling, are applied. The execution-cache-memory model is extended appropriately to accommodate not only modern Intel multicore chips but also the Intel Xeon Phi Knights Corner' coprocessor and an IBM POWER8 CPU. This allows us to discuss the impact of processor features on the performance across four modern architectures that are relevant for high performance computing. Copyright (c) 2016 John Wiley & Sons, Ltd.
Keywords: ECM performance model; kahan; scalar product; XEON; knights corner; POWER8

Performance analysis of the Kahan-enhanced scalar product on current multi-corecore and many-core processors - Hofmann, Johannes; Fey, Dietmar; Riedman, Michael; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Concurrency and Computation-Practice & Experience, 2016)
We investigate the performance characteristics of a numerically enhanced scalar product (dot) kernel loop that uses the Kahan algorithm to compensate for numerical errors, and describe efficient single instruction multiple data-vectorized implementations on recent multi-core and many-core processors. Using low-level instruction analysis and the execution-cache-memory performance model, we pinpoint the relevant performance bottlenecks for single-core and thread-parallel execution and predict performance and saturation behavior. We show that the Kahan-enhanced scalar product comes at almost no additional cost compared with the naive (non-Kahan) scalar product if appropriate low-level optimizations, notably single instruction multiple data vectorization and unrolling, are applied. The execution-cache-memory model is extended appropriately to accommodate not only modern Intel multicore chips but also the Intel Xeon Phi 'Knights Corner' coprocessor and an IBM POWER8 CPU. This allows us to discuss the impact of processor features on the performance across four modern architectures that are relevant for high performance computing.
UnivIS-Import:2017-01-09:Pub.2016.tech.IMMD.IMMD3.perfor_9
Keywords: ECM performance model; kahan; scalar product; XEON; knights corner; POWER8

Performance analysis of the Kahan-enhanced scalar product on current multicore processors - Hofmann, Johannes; Fey, Dietmar; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Accepted for PPAM 2015, 2015)
UnivIS-Import:2016-06-01:Pub.2015.tech.IMMD.phleir.perfor_8

Performance Engineering for a Medical Imaging Application on the Intel Xeon Phi Accelerator - Hofmann, Johannes; Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (ARCS Workshops'14, 2014)
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD3.perfor_5

Performance engineering for a tall & skinny matrix multiplication kernels on GPUs - Ernst, Dominik; Hager, Georg; Thies, Jonas; Wellein, Gerhard (Lecture Notes in Computer Science book series (LNCS, volume 12043), 2020)
Keywords: GPU; Tall & skinny; Matrix multiplication

Performance engineering for real and complex tall & skinny matrix multiplication kernels on GPUs - Ernst, Dominik; Hager, Georg; Thies, Jonas; Wellein, Gerhard (International Journal of High Performance Computing Applications, 2020)
General matrix-matrix multiplications with double-precision real and complex entries (DGEMM and ZGEMM) in vendor-supplied BLAS libraries are best optimized for square matrices but often show bad performance for tall & skinny matrices, which are much taller than wide. NVIDIA’s current CUBLAS implementation delivers only a fraction of the potential performance as indicated by the roofline model in this case. We describe the challenges and key characteristics of an implementation that can achieve close to optimal performance. We further evaluate different strategies of parallelization and thread distribution and devise a flexible, configurable mapping scheme. To ensure flexibility and allow for highly tailored implementations we use code generation combined with autotuning. For a large range of matrix sizes in the domain of interest we achieve at least 2/3 of the roofline performance and often substantially outperform state-of-the art CUBLAS results on an NVIDIA Volta GPGPU.
CRIS-Team Scopus Importer:2020-10-23
Keywords: complex; CUDA; GPU; matrix multiplication; Performance engineering; tall & skinny

Performance Engineering for the Lattice Boltzmann Method on GPGPUs: Architectural Requirements and Performance Results - Habich, Johannes; Feichtinger, Christian; Köstler, Harald; Hager, Georg; Wellein, Gerhard (Computers & Fluids, 2012)
UnivIS-Import:2015-03-09:Pub.2012.tech.IMMD.lsinfs._perfo

Performance Engineering of the Kernel Polynomal Method on Large-Scale CPU-GPU Systems - Kreutzer, Moritz; Hager, Georg; Wellein, Gerhard; Alvermann, Andreas; Fehske, Holger; Pieper, Andreas (Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2015)
The Kernel Polynomial Method (KPM) is a well-established scheme in quantum physics and quantum chemistry to determine the eigenvalue density and spectral properties of large sparse matrices. In this work we demonstrate the high optimization potential and feasibility of peta-scale heterogeneous CPU-GPU implementations of the KPM. At the node level we show that it is possible to decouple the sparse matrix problem posed by KPM from main memory bandwidth both on CPU and GPU. To alleviate the effects of scattered data access we combine loosely coupled outer iterations with tightly coupled block sparse matrix multiple vector operations, which enables pure data streaming. All optimizations are guided by a performance analysis and modelling process that indicates how the computational bottlenecks change with each optimization step. Finally we use the optimized node-level KPM with a hybrid-parallel framework to perform large scale heterogeneous electronic structure calculations for novel topological materials on a petascale-class Cray XC30 system.
UnivIS-Import:2016-06-01:Pub.2015.tech.IMMD.phleir.perfor
Keywords: Parallel programming; Quantum mechanics; Performance analysis; Sparse matrices

Performance Modeling of Streaming Kernels and Sparse Matrix-Vector Multiplication on A64FX - Alappat, Christie; Laukemann, Jan; Gruber, Thomas; Hager, Georg; Wellein, Gerhard; Meyer, Nils; Wettig, Tilo (Proceedings of PMBS 2020: Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, 2020)
The A64FX CPU powers the current #1 supercomputer on the Top500 list. Although it is a traditional cache-based multicore processor, its peak performance and memory bandwidth rival accelerator devices. Generating efficient code for such a new architecture requires a good understanding of its performance features. Using these features, we construct the Execution-Cache-Memory (ECM) performance model for the A64FX processor in the FX700 supercomputer and validate it using streaming loops. We also identify architectural peculiarities and derive optimization hints. Applying the ECM model to sparse matrix-vector multiplication (SpMV), we motivate why the CRS matrix storage format is inappropriate and how the SELL-C-σ format with suitable code optimizations can achieve bandwidth saturation for SpMV.
CRIS-Team Scopus Importer:2021-04-19
Keywords: A64FX; ECM model; sparse matrix-vector multiplication

Performance of Scientific Applications on Modern Supercomputers - Deserno, Frank; Hager, Georg; Brechtefeld, Frank; Wellein, Gerhard (High Performance Computing in Science and Engineering Munich 2004 Transactions of the Second Joint HLRB and KONWIHR Status and ResultWorkshop, March 2-3, 2004, Technical University of Munich, andLeibniz-Rechenzentrum Munich, Germany., 2005)
UnivIS-Import:2015-04-16:Pub.2005.tech.IMMD.lsinfs.perfor_0

Performance Patterns and Hardware Metrics on Modern Multicore Processors: Best Practices for Performance Engineering - Eitzinger, Jan; Hager, Georg; Wellein, Gerhard (Euro-Par 2012, 2012)
UnivIS-Import:2015-04-16:Pub.2012.tech.IMMD.phleir.perfor

PGAS implementation of SpMVM and LBM with GPI - Shahzad, Faisal; Wittmann, Markus; Kreutzer, Moritz; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Proceedings of the 7th International Conference on PGAS Programming Models, 2013)
UnivIS-Import:2015-04-16:Pub.2013.tech.IMMD.phleir.pgasim

PHIST: A Pipelined, Hybrid-Parallel Iterative Solver Toolkit - Thies, Jonas; Röhrig-Zöllner, Melven; Overmars, Nigel; Basermann, Achim; Ernst, Dominik; Hager, Georg; Wellein, Gerhard (Acm Transactions on Mathematical Software, 2020)
The increasing complexity of hardware and software environments in high-performance computing poses big challenges on the development of sustainable and hardware-efficient numerical software. This article addresses these challenges in the context of sparse solvers. Existing solutions typically target sustainability, flexibility, or performance, but rarely all of them. Our new library PHIST provides implementations of solvers for sparse linear systems and eigenvalue problems. It is a productivity platform for performance-aware developers of algorithms and application software with abstractions that do not obscure the view on hardware-software interaction. The PHIST software architecture and the PHIST development process were designed to overcome shortcomings of existing packages. An interface layer for basic sparse linear algebra functionality that can be provided by multiple backends ensures sustainability, and PHIST supports common techniques for improving scalability and performance of algorithms such as blocking and kernel fusion. We showcase these concepts using the PHIST implementation of a block Jacobi-Davidson solver for non-Hermitian and generalized eigenproblems. We study its performance on a multi-core CPU, a GPU, and a large-scale many-core system. Furthermore, we show how an existing implementation of a block Krylov-Schur method in the Trilinos package Anasazi can benefit from the performance engineering techniques used in PHIST.
CRIS-Team Scopus Importer:2020-12-11
Keywords: GPU computing; hybrid parallel computing; Jacobi-Davidson; performance engineering; Sparse eigenvalue solvers

Physical Oscillator Model for Supercomputing - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (14th IEEE/ACM Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS23), 2023)

Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (Proceedings - IEEE International Conference on Cluster Computing, ICCC, 2019)
Analytic, first-principles performance modeling of distributed-memory applications is difficult due to a wide spectrum of random disturbances caused by the application and the system. These disturbances (commonly called 'noise') run contrary to the assumptions about regularity that one usually employs when constructing simple analytic models. Despite numerous efforts to quantify, categorize, and reduce such effects, a comprehensive quantitative understanding of their performance impact is not available, especially for long, one-off delays of execution periods that have global consequences for the parallel application. In this work, we investigate various traces collected from synthetic benchmarks that mimic real applications on simulated and real message-passing systems in order to pin-point the mechanisms behind delay propagation. We analyze the dependence of the propagation speed of 'idle waves, ' i.e., propagating phases of inactivity, emanating from injected delays with respect to the execution and communication properties of the application, study how such delays decay under increased noise levels, and how they interact with each other. We also show how fine-grained noise can make a system immune against the adverse effects of propagating idle waves. Our results contribute to a better understanding of the collective phenomena that manifest themselves in distributed-memory parallel applications.
CRIS-Team Scopus Importer:2019-11-29

Pseudo-Vectorization and RISC Optimization Techniques for the Hitachi SR8000 Architecture - Hager, Georg; Deserno, Frank; Wellein, Gerhard (High Performance Computing in Science and Engineering, Munich 2002: Transactions of the First Joint HLRB and KONWIHR Status and Result Workshop, October 10-11, Technical University of Munich, Germany., 2003)
UnivIS-Import:2015-04-16:Pub.2003.zwiss.rrze.zensys.pseudo_1

Pushing the limits for medical image reconstruction on recent standard multicore processors - Eitzinger, Jan; Hager, Georg; Hofmann, Hannes; Hornegger, Joachim; Wellein, Gerhard (International Journal of High Performance Computing Applications, 2013)
Volume reconstruction by backprojection is the computational bottleneck in many interventional clinical computed tomography (CT) applications. Today vendors in this field replace special purpose hardware accelerators with standard hardware such as multicore chips and GPGPUs. Medical imaging algorithms are on the verge of employing high-performance computing (HPC) technology, and are therefore an interesting new candidate for optimization. This paper presents low-level optimizations for the backprojection algorithm, guided by a thorough performance analysis on four generations of Intel multicore processors (Harpertown, Westmere, Westmere EX, and Sandy Bridge). We choose the RabbitCT benchmark, a standardized testcase well supported in industry, to ensure transparent and comparable results. Our aim is to provide not only the fastest possible implementation but also compare with performance models and hardware counter data in order to fully understand the results. We separate the influence of algorithmic optimizations, parallelization, SIMD vectorization, and microarchitectural issues and pinpoint problems with current SIMD instruction set extensions on standard CPUs (SSE, AVX). The use of assembly language is mandatory for best performance. Finally, we compare our results to the best GPGPU implementations available for this open competition benchmark. © The Author(s) 2012.

Quantifying performance bottlenecks of stencil computations using the Execution-Cache-Memory model - Eitzinger, Jan; Hager, Georg; Wellein, Gerhard; Stengel, Holger (Association for Computing Machinery, 2014)
Stencil algorithms on regular lattices appear in many fields of computational science, and much effort has been put into optimized implementations. Such activities are usually not guided by performance models that provide estimates of expected speedup. Understanding the performance properties and bottlenecks by performance modeling enables a clear view on promising optimization opportunities. In this work we refine the recently developed Execution-Cache-Memory (ECM) model and use it to quantify the performance bottlenecks of stencil algorithms on a contemporary Intel processor. This includes applying the model to arrive at single-core performance and scalability predictions for typical "corner case" stencil loop kernels. Guided by the ECM model we accurately quantify the significance of "layer conditions," which are required to estimate the data traffic through the memory hierarchy, and study the impact of typical optimization approaches such as spatial blocking, strength reduction, and temporal blocking for their expected benefits. We also compare the ECM model to the widely known Roofline model.
UnivIS-Import:2015-04-14:Pub.2014.tech.IMMD.phleir.quanti
Keywords: Multicore; Optimization; Performance model; Stencils

RZBENCH: performance evaluation of current HPC architectures using low-level and application benchmarks - Hager, Georg; Stengel, Holger; Zeiser, Thomas; Wellein, Gerhard (High Performance Computing in Science and Engineering, Garching/Munich 2007: Transactions of the Third Joint HLRB and KONWIHR Status and Result Workshop, Dec. 3-4, 2007, Leibniz Supercomputing Centre, Garching/Munich, Germany, 2009)
UnivIS-Import:2015-04-20:Pub.2009.tech.IMMD.phleir._rzben

Selecting an Appropriate Computational Platform for Supporting the Development of New Catalyst Carriers - Zeiser, Thomas; Hager, Georg; Wellein, Gerhard; Inayat, Amer; Schwieger, Wilhelm; Heidig, Tobias; Freund, Hannsjörg (Innovatives Supercomputing in Deutschland : inSiDE, 2009)
UnivIS-Import:2015-03-09:Pub.2009.tech.IMMD.phleir._selec

SPEChpc 2021 Benchmarks on Ice Lake and Sapphire Rapids Infiniband Clusters: A Performance and Energy Case Study - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (14th IEEE/ACM Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS23), 2023)
In this work, fundamental performance, power, and energy characteristics of the full SPEChpc 2021 benchmark suite are assessed on two different clusters based on Intel Ice Lake and Sapphire Rapids CPUs using the MPI-only codes' variants. We use memory bandwidth, data volume, and scalability metrics in order to categorize the benchmarks and pinpoint relevant performance and scalability bottlenecks on the node and cluster levels. Common patterns such as memory bandwidth limitation, dominating communication and synchronization overhead, MPI serialization, superlinear scaling, and alignment issues could be identified, in isolation or in combination, showing that SPEChpc 2021 is representative of many HPC workloads. Power dissipation and energy measurements indicate that the modern Intel server CPUs have such a high idle power level that race-to-idle is the paramount strategy for energy to solution and energy-delay product minimization. On the chip level, only memory-bound code shows a clear advantage of Sapphire Rapids compared to Ice Lake in terms of energy to solution.<br

Speeding up a Lattice Boltzmann Kernel on nVIDIA GPUs - Habich, Johannes; Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Proceedings of the First International Conference on Parallel, Distributed and Grid Computing for Engineering, 2009)
This paper presents implementation strategies and optimization approaches for a lattice Boltzmann flow solver on nVIDIA graphics processing units (GPUs). We outline the GPU-specific implementation of the flow solver, potential problems that occur and discuss how to avoid or at least ameliorate them. It is shown that ten times the performance of a quad core based two socket server can be achieved with a single graphics card. Furthermore we employ STREAM benchmarks to evaluate achievable performance and to establish programming guidelines for applications. With regard to hybrid computing the attainable bandwidth of different interfaces involved is evaluated. © Civil-Comp Press, 2009.
UnivIS-Import:2015-04-16:Pub.2009.tech.IMMD.phleir._speed
Keywords: CFD; CUDA; GPU; HPC; Lattice Boltzmann; STREAM benchmarks

Taming the Bandwidth Behemoth. First Experiences on a Large SGI Altix System - Hager, Georg; Bergen, Benjamin; Lammers, Peter; Wellein, Gerhard (Innovatives Supercomputing in Deutschland : inSiDE, 2005)
UnivIS-Import:2015-03-09:Pub.2005.tech.IMMD.lsinfs.taming

The Role of Idle Waves, Desynchronization, and Bottleneck Evasion in the Performance of Parallel Programs - Afzal, Ayesha; Hager, Georg; Wellein, Gerhard (IEEE Transactions on Parallel and Distributed Systems, 2022)
The performance of highly parallel applications on distributed-memory systems is influenced by many factors. Analytic performance modeling techniques aim to provide insight into performance limitations and are often the starting point of optimization efforts. However, coupling analytic models across the system hierarchy (socket, node, network) fails to encompass the intricate interplay between the program code and the hardware, especially when execution and communication bottlenecks are involved. In this paper we investigate the effect of bottleneck evasion and how it can lead to automatic overlap of communication overhead with computation. Bottleneck evasion leads to a gradual loss of the initial bulk-synchronous behavior of a parallel code so that its processes become desynchronized. This occurs most prominently in memory-bound programs, which is why we choose memory-bound benchmark and application codes, specifically an MPI-augmented STREAM Triad, sparse matrix-vector multiplication, and a collective-avoiding Chebyshev filter diagonalization code to demonstrate the consequences of desynchronization on two different supercomputing platforms. We investigate the role of idle waves as possible triggers for desynchronization and show the impact of automatic asynchronous communication for a spectrum of code properties and parameters, such as saturation point, matrix structures, domain decomposition, and communication concurrency. Our findings reveal how eliminating synchronization points (such as collective communication or barriers) precipitates performance improvements that go beyond what can be expected by simply subtracting the overhead of the collective from the overall runtime.
CRIS-Team Scopus Importer:2022-11-25
Keywords: Bandwidth; Bars; bottleneck; Codes; Computational modeling; Delays; desynchronization; Parallel distributed computing; performance modeling; performance optimization; scalability; synchronization; Synchronization; Topology

The world's fastest CPU and SMP node: Some performance results from the NEC SX-9 - Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (Proceedings of the IEEE International Symposium on Parallel&Distributed Processing 2009, 2009)
Classic vector systems have all but vanished from recent TOP500 lists. Looking at the newly introduced NEC SX-9 series, we benchmark its memory subsystem using the low level vector triad and employ an advanced lattice Boltzmann flow solver kernel to demonstrate that classic vectors still combine excellent performance with a well-established optimization approach. Results for commodity x86-based systems are provided for reference. © 2009 IEEE.
UnivIS-Import:2015-04-16:Pub.2009.tech.IMMD.phleir._thewo

Towards an exascale enabled sparse solver repository - Thies, Jonas; Galgon, Martin; Shahzad, Faisal; Alvermann, Andreas; Kreutzer, Moritz; Pieper, Andreas; Röhrig-Zöllner, Melven; Basermann, Achim; Fehske, Holger; Hager, Georg; Lang, Bruno; Wellein, Gerhard (Springer Verlag, 2016)
As we approach the exascale computing era, disruptive changes in the software landscape are required to tackle the challenges posed by manycore CPUs and accelerators. We discuss the development of a new ‘exascale enabled’ sparse solver repository (the ESSR) that addresses these challenges—from fundamental design considerations and development processes to actual implementations of some prototypical iterative schemes for computing eigenvalues of sparse matrices. Key features of the ESSR include holistic performance engineering, tight integration between software layers and mechanisms to mitigate hardware failures.

Towards optimal performance for lattice Boltzmann applications on terascale computers - Wellein, Gerhard; Lammers, Peter; Hager, Georg; Donath, Stefan; Zeiser, Thomas (Parallel Computational Fluid Dynamics: Theory and Applications, Proceedings of the 2005 International Conference on Parallel Computational Fluid Dynamics, 2006)
The chapter discusses the lattice Boltzmann method (LBM) performance on commodity "off-the-shelf" clusters with Intel Xeon processors, tailored HPC systems, and a NEC SX8 vector system. The chapter describes the main architectural differences and comments on single processor performance as well as optimization strategies. The parallel performance of a large scale simulation running on up to 2000 processors, providing 2 TFlop/s of sustained performance is evaluated and presented in the chapter. In the past decade, the LBM has been established as an alternative for the numerical simulation of incompressible flows. One major reason for the success of LBM is the simplicity of its core algorithm that allows both easy adaption to complex application scenarios as well as extension to additional physical or chemical effects. Because LBM is a direct method, the use of extensive computer resources is often mandatory. Thus, LBM has attracted a lot of attention in the high-performance computing (HFC) community. An important feature of many LBM codes is that the core algorithm can be reduced to a few manageable subroutines, facilitating deep performance analysis followed by precise code and data layout optimization. © 2006 Elsevier B.V. All rights reserved.
UnivIS-Import:2015-04-16:Pub.2006.tech.IMMD.lsinfs._towar

Understanding HPC benchmark performance on intel broadwell and cascade lake processors - Alappat, Christie; Hofmann, Johannes; Hager, Georg; Fehske, Holger; Bishop, Alan R.; Wellein, Gerhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
Hardware platforms in high performance computing are constantly getting more complex to handle even when considering multicore CPUs alone. Numerous features and configuration options in the hardware and the software environment that are relevant for performance are not even known to most application users or developers. Microbenchmarks, i.e., simple codes that fathom a particular aspect of the hardware, can help to shed light on such issues, but only if they are well understood and if the results can be reconciled with known facts or performance models. The insight gained from microbenchmarks may then be applied to real applications for performance analysis or optimization. In this paper we investigate two modern Intel x86 server CPU architectures in depth: Broadwell EP and Cascade Lake SP. We highlight relevant hardware configuration settings that can have a decisive impact on code performance and show how to properly measure on-chip and off-chip data transfer bandwidths. The new victim L3 cache of Cascade Lake and its advanced replacement policy receive due attention. Finally we use DGEMM, sparse matrix-vector multiplication, and the HPCG benchmark to make a connection to relevant application scenarios.
CRIS-Team Scopus Importer:2020-07-10
Keywords: Benchmarking; Intel; Microbenchmarking; x86

Vector Computers in a World of Commodity Clusters, Massively Parallel Systems and Many-Core Many-Threaded CPUs: Recent Experience Based on an Advanced Lattice Boltzmann Flow Solver - Zeiser, Thomas; Hager, Georg; Wellein, Gerhard (High Performance Computing in Science and Engineering '08: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2008, 2009)
This report summarizes experience gained during the last year using the NEC SX-8 at HLRS and its wide range of competitors: commodity clusters with Infiniband interconnect, massively parallel systems (Cray XT4, IBM BlueGene L/P) and emerging many-core many-threaded CPUs (SUN Niagara2 processor). The observations are based on low-level benchmarks and the use of an advanced lattice Boltzmann flow solver developed in the framework of an international development consortium (ILBDC).
UnivIS-Import:2015-04-20:Pub.2009.tech.IMMD.phleir._vecto

What's next? Evaluating Performance and Programming Approaches for Emerging Computer Technologies - Wellein, Gerhard; Hager, Georg; Rüde, Ulrich (2008)
UnivIS-Import:2015-03-09:Pub.2008.tech.IMMD.lsinfs.whatsn

YaskSite: Stencil Optimization Techniques Applied to Explicit ODE Methods on Modern Architectures - Alappat, Christie; Seiferth, Johannes; Hager, Georg; Korch, Matthias; Rauber, Thomas; Wellein, Gerhard (CGO 2021 - Proceedings of the 2021 IEEE/ACM International Symposium on Code Generation and Optimization, 2021)
The landscape of multi-core architectures is growing more complex and diverse. Optimal application performance tuning parameters can vary widely across CPUs, and finding them in a possibly multidimensional parameter search space can be time consuming, expensive and potentially infeasible. In this work, we introduce YaskSite, a tool capable of tackling these challenges for stencil computations. YaskSite is built upon Intel's YASK framework. It combines YASK's flexibility to deal with different target architectures with the Execution-Cache-Memory performance model, which enables identifying optimal performance parameters analytically without the need to run the code. Further we show that YaskSite's features can be exploited by external tuning frameworks to reliably select the most efficient kernel(s) for the application at hand. To demonstrate this, we integrate YaskSite into Offsite, an offline tuner for explicit ordinary differential equation methods, and show that the generated performance predictions are reliable and accurate, leading to considerable performance gains at minimal code generation time and autotuning costs on the latest Intel Cascade Lake and AMD Rome CPUs.
CRIS-Team Scopus Importer:2021-04-19
Keywords: Stencil optimization; YASK; ECM model; Autotuning; PIRK methods; Performance modeling