A Local Parallel Communication Algorithm for Polydisperse Rigid Body Dynamics - Eibl, Sebastian; Rüde, Ulrich (Parallel Computing, 2018)
The simulation of large ensembles of particles is usually parallelized by partitioning the domain spatially and using message passing to communicate between the processes handling neighboring subdomains. The particles are represented as individual geometric objects and are assigned to the subdomains. Handling collisions and migrating particles between subdomains, as required for proper parallel execution, requires a complex communication protocol. Typically, the parallelization is restricted to handling only particles that are smaller than a subdomain. In many applications, however, particle sizes may vary drastically with some of them being larger than a subdomain. In this article we propose a new communication and synchronization algorithm that can handle the parallelization without size restrictions on the particles. Despite the additional complexity and extended functionality, the new algorithm introduces only minimal overhead. We demonstrate the scalability of the previous and the new communication algorithms up to almost two million parallel processes and for handling ten billion (1010) geometrically resolved particles on a state-of-the-art petascale supercomputer. Different scenarios are presented to analyze the performance of the new algorithm and to demonstrate its capability to simulate polydisperse scenarios, where large individual particles can extend across several subdomains.
EAM Import::2019-03-13

A MASSIVELY PARALLEL EULERIAN-LAGRANGIAN METHOD FOR ADVECTION-DOMINATED TRANSPORT IN VISCOUS FLUIDS - Kohl, Nils; Mohr, Marcus; Eibl, Sebastian; Rüde, Ulrich (SIAM Journal on Scientific Computing, 2022)
Motivated by challenges in the Earth’s mantle convection, we present a massively parallel implementation of an Eulerian-Lagrangian method for the advection-diffusion equation in the advection-dominated regime. The advection term is treated by a particle-based characteristics method coupled to a block-structured finite element framework. Its numerical and computational performance is evaluated in multiple two- and three-dimensional benchmarks, including curved geometries, discontinuous solutions, and pure advection, and it is applied to a coupled nonlinear system modeling buoyancy-driven convection in Stokes flow. We demonstrate the parallel performance in a strong and weak scaling experiment, with scalability to up to 147,456 parallel processes, solving for more than 5.2 × 1010 (52 billion) degrees of freedom per time-step.
CRIS-Team Scopus Importer:2022-08-12
Keywords: advection-diffusion; Eulerian-Lagrangian methods; parallel algorithms

A Systematic Comparison of Dynamic Load Balancing Algorithms for Massively Parallel Rigid Particle Dynamics - Eibl, Sebastian; Rüde, Ulrich (arXiv, 2018)
As compute power increases with time, more involved and larger simulations become possible. However, it gets increasingly difficult to efficiently use the provided computational resources. Especially in particle-based simulations with a spatial domain partitioning large load imbalances can occur due to the simulation being dynamic. Then a static domain partitioning may not be suitable. This can deteriorate the overall runtime of the simulation significantly. Sophisticated load balancing strategies must be designed to alleviate this problem. In this paper we conduct a systematic evaluation of the performance of six different load balancing algorithms. Our tests cover a wide range of simulation sizes, and employ one of the largest supercomputers available. In particular we study the runtime and memory complexity of all components of the simulation carefully. When progressing to extreme scale simulations it is essential to identify bottlenecks and to predict the scaling behaviour. Scaling experiments are shown for up to over one million processes. The performance of each algorithm is analyzed with respect to the quality of the load balancing and its runtime costs. For all tests, the waLBerla multiphysics framework is employed.<br

A systematic comparison of runtime load balancing algorithms for massively parallel rigid particle dynamics - Eibl, Sebastian; Rüde, Ulrich (Computer Physics Communications, 2019)
Keywords: Discrete element method; Rigid body dynamics; Non-smooth granular dynamics; Runtime load balancing; Runtime domain partitioning

Comparative Study of the Rheological Behavior of Sheared Bi- and Polydisperse Sediment Beds via Fully Resolved Simulations - Rettinger, Christoph; Eibl, Sebastian; Rüde, Ulrich; Konidena, Sudarshan; Vowinckel, Bernhard (39th IAHR World Congress, 2022)

Densification of single-walled carbon nanotube films: Mesoscopic distinct element method simulations and experimental validation - Drozdov, Grigorii; Ostanin, Igor; Xu, Hao; Wang, Yuezhou; Dumitricǎ, Traian; Grebenko, Artem; Tsapenko, Alexey P.; Gladush, Yuriy; Ermolaev, Georgy; Volkov, Valentyn S.; Eibl, Sebastian; Rüde, Ulrich; et al. (Journal of Applied Physics, 2020)
Nanometer-thin single-walled carbon nanotube (CNT) films collected from the aerosol chemical deposition reactors have gathered attention for their promising applications. Densification of these pristine films provides an important way to manipulate mechanical, electronic, and optical properties. To elucidate the underlying microstructural level restructuring, which is ultimately responsible for the change in properties, we perform large scale vector-based mesoscopic distinct element method simulations in conjunction with electron microscopy and spectroscopic ellipsometry characterization of pristine and densified films by drop-cast volatile liquid processing. Matching with the microscopy observations, pristine CNT films with a finite thickness are modeled as self-assembled CNT networks comprising entangled dendritic bundles with branches extending down to individual CNTs. Simulations of these films under uniaxial compression uncover a soft deformation regime extending up to an ∼75% strain. When removing the loads, the pre-compressed samples evolve into homogeneously densified films with thickness values depending on both the pre-compression level and the sample microstructure. The significant reduction in thickness is attributed to the underlying structural changes occurring at the 100 nm scale, including the zipping of the thinnest dendritic branches.
CRIS-Team Scopus Importer:2020-11-27

Dynamic Load Balancing for Large Scale Particle Simulations - Eibl, Sebastian; Schornbaum, Florian; Rüde, Ulrich (The 5th Joint International Conference on Multibody System Dynamics, 2018)
Current and future applications of rigid body dynamics involve an ever growing number of particles. Example applications are in the pharmaceutical industries, chemical industries or additive manufacturing. Simulations of billions of particles can only be performed by using specialized software. This software must use today’s largest supercomputers efficiently. In these highly parallel environments special attention must be paid to the load balance between the processes. Since the overall performance is dependent on the slowest process overloaded processes will lead to a decrease of the performance. Typically rigid body dynamics algorithms are parallelized by partition- ing the domain spatially into subdomains. These are subsequently assigned to processes. Each process evolves the particles contained in its subdomain(s) in time. The domain partitioning must be chosen carefully. The workload associated with each subdomain is related to the particles within. Deciding on the domain partitioning at the begin- ning of a simulation and using it throughout the whole simulation works well for homogeneous setups where the particle density does not fluctuate much. But typical rigid body dynamics simulations involve a huge number of moving particles. This may change the workload of the subdomains constantly throughout the simulation. There- fore a static domain partitioning that achieved a good load balance at the beginning of the simulation might not be appropriate later on. To guarantee good performance over the whole simulation the software framework needs to dynamically rebalance the workload. It has to update its domain partitioning and redistribute it at regular intervals. This allows to avoid unbalanced workloads. Many popular software packages like LIGGGHTS, DEMOOP, ls1 mardyn already incorporate various load balancing strategies [1, 2, 3]. For fluid dynamics simulations good scaling behaviour has already been reported for diffusion based load balancing algorithms [4, 5]. In this work, we study the applicability of load balancing algorithms based on space filling curves [6] and diffusion based algorithms [7] for rigid body dynamics simulations. We investigate the scaling behaviour of these algorithms on the state-of-the-art supercomputer Juqueen (TOP500 rank: 21) located at the Jlich Supercomputing Centre. This supercomputer allows simulations with more than one million processes which we use to determine the usability of these algorithms for future large scale simulations. Preliminary results for Hilbert space filling curves presented in Fig. 1 show suboptimal scaling behaviour. The performance of the load balancing starts to drop significantly above 8192 processes which prohibits scaling up to the full machine (1.8 million processes). Due to inherent global gather operations needed for space filling curves performance decreases for a large number of processes. We will also try to reproduce the results obtained for diffusive load balancing algorithms in fluid dynamics [5] in rigid body dynamics. References [1] R. Berger, C. Kloss, A. Kohlmeyer, and S. Pirker, “Hybrid parallelization of the LIGGGHTS open-source DEM code,” Powder Technology, vol. 278, pp. 234–247, July 2015. [2] D. T. Cintra, R. B. Willmersdorf, P. R. M. Lyra, and W. W. M. Lira, “A hybrid parallel DEM approach with workload balancing based on HSFC,” Engineering Computations, vol. 33, pp. 2264–2287, Nov. 2016. [3] A. Heinecke, W. Eckhardt, M. Horsch, and H.-J. Bungartz, Supercomputing for Molecular Dynamics Simulations: Han- dling Multi-Trillion Particles in Nanofluidics. Springer, 2015. [4] F. Schornbaum and U. Rüde, “Massively parallel algorithms for the lattice boltzmann method on NonUniform grids,” SIAM Journal on Scientific Computing, vol. 38, pp. C96–C126, Jan. 2016. [5] F. Schornbaum and U. Rüde, “Extreme-scale block-structured adaptive mesh refinement,” arXiv preprint arXiv:1704.06829, 2017. [6] P. M. Campbell, K. D. Devine, J. E. Flaherty, L. G. Gervasio, and J. D. Teresco, “Dynamic octree load balancing using space-filling curves,” Williams College Department of Computer Science, Tech. Rep. CS-03-01, 2003. [7] G. Cybenko, “Dynamic load balancing for distributed memory multiprocessors,” Journal of Parallel and Distributed Computing, vol. 7, pp. 279–301, Oct. 1989. <br

Fully Resolved Simulations of Dune Formation in Riverbeds - Rettinger, Christoph; Godenschwager, Christian; Eibl, Sebastian; Preclik, Tobias; Schruff, Tobias; Frings, Roy M.; Rüde, Ulrich (High Performance Computing: 32nd International Conference, ISC High Performance 2017, Frankfurt, Germany, June 18-22, 2017, Proceedings, 2017)

JUQUEEN Extreme Scaling Workshop 2017 - Eibl, Sebastian; Preclik, Tobias; Rüde, Ulrich (2017)
From 23 to 25 January 2017, JSC organised its eighth IBM Blue Gene Extreme Scaling Workshop. The entire 28-rack JUQUEEN Blue Gene/Q was reserved for over 50 hours to allow six selected code teams to investigate and improve the scalability of their applications. Ultimately, all six codes managed to run using the full complement of 458,752 cores (most with over 1.8 million threads). MPAS-A (KIT/NCAR) and the pe rigid body physics engine (FAU) were both able to display strong scalability to 28 racks and thereby become candidates for High-Q Club membership. MPAS-A returned after participating in the 2015 workshop with a higher resolution dataset and substantially improved file I/O using SIONlib to successfully manage its largest ever global atmospheric simulation. While the hydrology simulator ParFlow (UBonn/FZJ-IGB) had recently demonstrated execution scaling to the full JUQUEEN without file writing enabled, during the workshop the focus was investigating file I/O performance which remains a bottleneck. High-Q Club member KKRnano (FZJ-IAS) investigated the scalability of a new solver algorithm developed to handle a million atoms, while the latest version of CPMD was tested with a large 1500-atom system. Both of these quantum materials codes uncovered performance limitations at larger scales. The final code was a prototype multi-compartmental neuronal network simulator, NestMC (JSC SimLab Neuroscience), which compared scalability of different threading implementations. Detailed reports are provided by each code-team, and additional comparative analysis to the 28 High-Q Club member code

Numerical study of sheared mobile polydisperse sediment beds with a coupled lattice Boltzmann-discrete element method - Rettinger, Christoph; Eibl, Sebastian; Rüde, Ulrich; Vowinckel, Bernhard (EGU General Assembly, 2021)
With the increasing computational power of today's supercomputers, geometrically fully resolved simulations of particle-laden flows are becoming a viable alternative to laboratory experiments. Such simulations enable detailed investigations of transport phenomena in various multiphysics scenarios, such as the coupled interaction of sediment beds with a shearing fluid flow. There, the majority of available simulations as well as experimental studies focuses on setups of monodisperse particles. In reality, however, polydisperse configurations are much more common and feature unique effects like vertical size segregation. In this talk, we will present numerical studies of mobile polydisperse sediment beds in a laminar shear flow, with a ratio of maximum to minimum diameter up to 10. The lattice Boltzmann method is applied to represent the fluid dynamics through and above the sediment bed efficiently. We model particle interactions by a discrete element method and explicitly account for lubrication forces. The fluid-particle coupling mechanism is based on the geometrically fully resolved momentum transfer between the fluid and the particulate phase. We will highlight algorithmic aspects and communication schemes essential for massively parallel execution. Utilizing these capabilities allows us to achieve large-scale simulations with more than 26.000 densely-packed polydisperse particles interacting with the fluid. With this, we are able to reproduce effects like size segregation and to study the rheological behavior of such systems in great detail. We will evaluate and discuss the influence of polydispersity on these processes. These insights will be used to improve and extend existing macroscopic model

Optimized Communication Scheme for Massively Parallel Simulations of Polydisperse Particle Systems with Large Size Ratios - Eibl, Sebastian; Preclik, Tobias; Rüde, Ulrich (Particles 2017, 2017)
The standard method to parallelize particle based simulations with a hard contact interaction model and impulsive contact reactions[1] is to subdivide the simulation domain into subdomains. Each subdomain is handled by a different process and particle related information is communicated to neighbouring domains. This next neighbour communication shows perfect scalability even for a very large number of processes[2]. However it imposes an upper limit for the particle size as information can not propagate to the subdomain after the next one. Making the subdomains larger is not an option as it limits the usage of supercomputers which require thousands of processes to run efficiently. One way to overcome this problem is to introduce global particles which are known on every process. This however requires all to all communication to keep the particles in sync which impedes good scaling performance on large systems. Here we propose a new communication scheme which can handle polydisperse particles of arbitrary size ratios. The largest particles are allowed to overlap more than one subdomain and are not restricted in size. The communication scheme is based on point to point communication and completely avoids expensive collective operations. Only processes which really have to know about a particle will participate in the communication corresponding to this particle. Aggressive message aggregation is used to reduce the amount of messages sent. Also an information caching system is applied to further reduce the number of messages. We implemented the communication scheme into our rigid body physics engine module pe of the waLBerla framework[3] and investigated the scaling behaviour on the Juqueen supercomputer in Jülich. We achieved perfect weak scalability up to the full machine. For the strong scaling the communication protocol converges to an all to all communication as large particles overlap more and more subdomains. Therefore as soon as this process starts the parallel efficiency deteriorates. REFERENCES [1] D. E. Stewart, Rigid-body dynamics with friction and impact. SIAM review 42.1 (2000): 3-39. [2] T. Preclik & U. Rüde, Ultrascale simulations of non-smooth granular dynamics., Computational Particle Mechanics 2 (2015), Nr. 2, S. 173–196 [3] C. Godenschwager et al., A framework for hybrid parallel flow simulations with a trillion cells in complex geometries, Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, 2013, 35 <br

pe - Rigid Body Physics Engine - Eibl, Sebastian; Preclik, Tobias; Rüde, Ulrich (JUQUEEN Extreme Scaling Workshop 2017, 2017)
UnivIS-Import:2017-12-18:Pub.2017.tech.IMMD.lsinfs.perigi_1

Resiliency in numerical algorithm design for extreme scale simulations - Agullo, Emmanuel; Altenbernd, Mirco; Anzt, Hartwig; Bautista-Gomez, Leonardo; Benacchio, Tommaso; Bonaventura, Luca; Bungartz, Hans-Joachim; Chatterjee, Sanjay; Ciorba, Florina M.; Debardeleben, Nathan; Drzisga, Daniel; Eibl, Sebastian; et al. (International Journal of High Performance Computing Applications, 2021)
This work is based on the seminar titled 'Resiliency in Numerical Algorithm Design for Extreme Scale Simulations' held March 1-6, 2020, at Schloss Dagstuhl, that was attended by all the authors. Advanced supercomputing is characterized by very high computation speeds at the cost of involving an enormous amount of resources and costs. A typical large-scale computation running for 48 h on a system consuming 20 MW, as predicted for exascale systems, would consume a million kWh, corresponding to about 100k Euro in energy cost for executing 10(23) floating-point operations. It is clearly unacceptable to lose the whole computation if any of the several million parallel processes fails during the execution. Moreover, if a single operation suffers from a bit-flip error, should the whole computation be declared invalid? What about the notion of reproducibility itself: should this core paradigm of science be revised and refined for results that are obtained by large-scale simulation? Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated. More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? While the analysis of use cases can help understand the particular reliability requirements, the construction of remedies is currently wide open. One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors. These ideas constituted an essential topic of the seminar. The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge. This article gathers a broad range of perspectives on the role of algorithms, applications and systems in achieving resilience for extreme scale simulations. The ultimate goal is to spark novel ideas and encourage the development of concrete solutions for achieving such resilience holistically.
CRIS-Team WoS Importer:2021-12-24

Rheology of mobile sediment beds in laminar shear flow: effects of creep and polydispersity - Rettinger, Christoph; Eibl, Sebastian; Rüde, Ulrich; Vowinckel, Bernhard (Journal of Fluid Mechanics, 2022)

Size-Independent Mechanical Response of Ultrathin Carbon Nanotube Films in Mesoscopic Distinct Element Method Simulations - Ostanin, Igor; Dumitrica, Traian; Eibl, Sebastian; Rüde, Ulrich (Journal of Applied Mechanics, 2019)
In this work, we present a computational study of the small strain mechanics of freestanding ultrathin carbon nanotube (CNT) films under in-plane loading. The numerical modeling of the mechanics of representatively large specimens with realistic micro- and nanostructure is presented. Our simulations utilize the scalable implementation of the mesoscopic distinct element method of the waLBerla multi-physics framework. Within our modeling approach, CNTs are represented as chains of interacting rigid segments. Neighboring segments in the chain are connected with elastic bonds, resolving tension, bending, shear, and torsional deformations. These bonds represent a covalent bonding within the CNT surface and utilize enhanced vector model (EVM) formalism. Segments of the neighboring CNTs interact with realistic coarse-grained anisotropic van der Waals potential, enabling a relative slip of CNTs in contact. The advanced simulation technique allowed us to gain useful insights on the behavior of CNT materials. It was established that the energy dissipation during CNT sliding leads to extended load transfer that conditions size-independent, material-like mechanical response of the weakly bonded assemblies of CNTs.
CRIS-Team WoS Importer:2020-01-21

The maximum dissipation principle in rigid-body dynamics with inelastic impacts - Preclik, Tobias; Eibl, Sebastian; Rüde, Ulrich (Computational Mechanics, 2017)
UnivIS-Import:2017-12-18:Pub.2017.tech.IMMD.lsinfs.themax
Keywords: Impulse; Collision; Rigid Body Dynamics; Contact Dynamics

tinyMD: Mapping molecular dynamics simulations to heterogeneous hardware using partial evaluation - Ravedutti Lucio Machado, Rafael; Schmitt, Jonas; Eibl, Sebastian; Eitzinger, Jan; Leißa, Roland; Hack, Sebastian; Pérard-Gayot, Arsène; Membarth, Richard; Köstler, Harald (Journal of Computational Science, 2021)
This paper investigates the suitability of the AnyDSL partial evaluation framework to implement tinyMD: an efficient, scalable, and portable simulation of pairwise interactions among particles. We compare tinyMD with the miniMD proxy application that scales very well on parallel supercomputers. We discuss the differences between both implementations and contrast miniMD's performance for single-node CPU and GPU targets, as well as its scalability on SuperMUC-NG and Piz Daint supercomputers. Additionally, we demonstrate tinyMD's flexibility by coupling it with the waLBerla multi-physics framework. This allow us to execute tinyMD simulations using the load-balancing mechanism implemented in waLBerla.
CRIS-Team Scopus Importer:2021-08-13
Keywords: High performance computing; Load balancing; Molecular dynamics; Partial evaluation

Toward large scale modeling of carbon nanotube systems with the mesoscopic distinct element method - Ostanin, I. A.; Zhilyaev, Petr; Petrov,; Dumitrica, T.; Eibl, Sebastian; Rüde, Ulrich; Kuzkin, V. A. (Letters on Materials, 2018)
Keywords: Carbon nanotubes; Distinct element method; Enhanced vector model; Parallel computing

WALBERLA: A block-structured high-performance framework for multiphysics simulations - Bauer, Martin; Eibl, Sebastian; Godenschwager, Christian; Kohl, Nils; Kuron, Michael; Rettinger, Christoph; Schornbaum, Florian; Schwarzmeier, Christoph; Thönnes, Dominik; Köstler, Harald; Rüde, Ulrich (Computers & Mathematics with Applications, 2020)
Programming current supercomputers efficiently is a challenging task. Multiple levels of parallelism on the core, on the compute node, and between nodes need to be exploited to make full use of the system. Heterogeneous hardware architectures with accelerators further complicate the development process. WALBERLA addresses these challenges by providing the user with highly efficient building blocks for developing simulations on block-structured grids. The block-structured domain partitioning is flexible enough to handle complex geometries, while the structured grid within each block allows for highly efficient implementations of stencil-based algorithms. We present several example applications realized with WALBERLA, ranging from lattice Boltzmann methods to rigid particle simulations. Most importantly, these methods can be coupled together, enabling multiphysics simulations. The framework uses meta-programming techniques to generate highly efficient code for CPUs and GPUs from a symbolic method formulation.
CRIS-Team Scopus Importer:2020-02-04
Keywords: Adaptive mesh refinement; Code generation; High-performance computing; Lattice Boltzmann; Multiphysics; Rigid particle dynamics