Visualization in the integrated SimPhoNy multiscale simulation framework
Abstract We describe three distinct approaches to visualization for multiscale materials modelling research. These have been developed with the framework of the SimPhoNy FP7 EU-project, and complement each other in their requirements and possibilities. All have been integrated via wrappers to one or more of the simulation approaches within the SimPhoNy project. In this manuscript we describe and contrast their features. Together they cover visualization needs from electronic to macroscopic scales and are suited to simulations made on personal computers, workstations or advanced High Performance parallel computers. Examples as well as recommendations for future calculations are presented.
Lattice Boltzmann Simulations at Petascale on Multi-GPU Systems with Asynchronous Data Transfer and Strictly Enforced Memory Read Alignment
The lattice Boltzmann method is a well-established numerical approach for complex fluid flow simulations. Recently general-purpose graphics processing units have become accessible as high-performance computing resources at large-scale. We report on implementing a lattice Boltzmann solver for multi-GPU systems that achieves 0.69 PFLOPS performance on 16384 GPUs. In addition to optimizing the data layout on the GPUs and eliminating the halo sites, we make use of the possibility to overlap data transfer between the host CPU and the device GPU with computing on the GPU. We simulate flow in porous media and measure both strong and weak scaling performance with the emphasis being on a large scale…
A boundary condition for arbitrary shaped inlets in lattice-Boltzmann simulations
We introduce a mass-flux-based inlet boundary condition for the lattice-Boltzmann method. The proposed boundary condition requires minimal amount of boundary data, it produces a steady-state velocity field which is accurate close to the inlet even for arbitrary inlet geometries, and yet it is simple to implement. We demonstrate its capability for both simple and complex inlet geometries by numerical experiments. For simple inlet geometries, we show that the boundary condition provides very accurate inlet velocities when Re less than or similar to 1. Even with moderate Reynolds number, the inlet velocities are accurate for practical purposes. Furthermore, the potential of our boundary condit…
Comparison of implementations of the lattice-Boltzmann method
AbstractSimplicity of coding is usually an appealing feature of the lattice-Boltzmann method (LBM). Conventional implementations of LBM are often based on the two-lattice or the two-step algorithm, which however suffer from high memory consumption and poor computational performance, respectively. The aim of this work was to identify implementations of LBM that would achieve high computational performance with low memory consumption. Effects of memory addressing schemes were investigated in particular. Data layouts for velocity distribution values were also considered, and they were found to be related to computational performance. A novel bundle data layout was therefore introduced. Address…
Simulation Software for Flow of Fluid with Suspended Point Particles in Complex Domains: Application to Matrix Diffusion
Matrix diffusion is a phenomenon in which tracer particles convected along a flow channel can diffuse into porous walls of the channel, and it causes a delay and broadening of the breakthrough curve of a tracer pulse. Analytical and numerical methods exist for modeling matrix diffusion, but there are still some features of this phenomenon, which are difficult to address using traditional approaches. To this end we propose to use the lattice-Boltzmann method with point-like tracer particles. These particles move in a continuous space, are advected by the flow, and there is a stochastic force causing them to diffuse. This approach can be extended to include particle-particle and particle-wall…
A prospect for computing in porous materials research: Very large fluid flow simulations
Abstract Properties of porous materials, abundant both in nature and industry, have broad influences on societies via, e.g. oil recovery, erosion, and propagation of pollutants. The internal structure of many porous materials involves multiple scales which hinders research on the relation between structure and transport properties: typically laboratory experiments cannot distinguish contributions from individual scales while computer simulations cannot capture multiple scales due to limited capabilities. Thus the question arises how large domain sizes can in fact be simulated with modern computers. This question is here addressed using a realistic test case; it is demonstrated that current …
Designing a graphics processing unit accelerated petaflop capable lattice Boltzmann solver: Read aligned data layouts and asynchronous communication
The lattice Boltzmann method is a well-established numerical approach for complex fluid flow simulations. Recently, general-purpose graphics processing units (GPUs) have become available as high-performance computing resources at large scale. We report on designing and implementing a lattice Boltzmann solver for multi-GPU systems that achieves 1.79 PFLOPS performance on 16,384 GPUs. To achieve this performance, we introduce a GPU compatible version of the so-called bundle data layout and eliminate the halo sites in order to improve data access alignment. Furthermore, we make use of the possibility to overlap data transfer between the host central processing unit and the device GPU with comp…
Implementation techniques for the lattice Boltzmann method
Mass-flux-based outlet boundary conditions for the lattice Boltzmann method
We present outlet boundary conditions for the lattice Boltzmann method. These boundary conditions are constructed with a mass-flux-based approach. Conceptually, the mass-flux-based approach provides a mathematical framework from which specific boundary conditions can be derived by enforcing given physical conditions. The object here is, in particular, to explain the mass-flux-based approach. Furthermore, we illustrate, transparently, how boundary conditions can be derived from the emerging mathematical framework. For this purpose, we derive and present explicitly three outlet boundary conditions. By construction, these boundary conditions have an apparent physical interpretation which is fu…
Modeling mass transfer in fracture flows with the time domain-random walk method
The time domain-random walk method was developed further for simulating mass transfer in fracture flows together with matrix diffusion in surrounding porous media. Specifically, a time domain-random walk scheme was developed for numerically approximating solutions of the advection-diffusion equation when the diffusion coefficient exhibits significant spatial variation or even discontinuities. The proposed scheme relies on second-order accurate, central-difference approximations of the advective and diffusive fluxes. The scheme was verified by comparing simulated results against analytical solutions in flow configurations involving a rectangular channel connected on one side with a porous ma…
Chromatin organization regulates viral egress dynamics.
Various types of DNA viruses are known to elicit the formation of a large nuclear viral replication compartment and marginalization of the cell chromatin. We used three-dimensional soft x-ray tomography, confocal and electron microscopy, combined with numerical modelling of capsid diffusion to analyse the molecular organization of chromatin in herpes simplex virus 1 infection and its effect on the transport of progeny viral capsids to the nuclear envelope. Our data showed that the formation of the viral replication compartment at late infection resulted in the enrichment of heterochromatin in the nuclear periphery accompanied by the compaction of chromatin. Random walk modelling of herpes s…
Using microtomography, image analysis and flow simulations to characterize soil surface seals
Raindrops that impact on soil surface affect the pore structure and form compact soil surface seals. Damaged pore structure reduces water infiltration which can lead to increased soil erosion. We introduce here methods to characterize the properties of surface seals in a detailed manner. These methods include rainfall simulations, x-ray microtomography, image analysis and pore-scale flow simulations. Methods were tested using clay soil samples, and the results indicate that the sealing process changes several properties of the pore structure.
Coupling of lattice-Boltzmann solvers with suspended particles using the MPI intercommunication framework
Abstract The MPI intercommunication framework was used for coupling of two lattice-Boltzmann solvers with suspended particles, which model advection and diffusion respectively of these particles in a carrier fluid. Simulation domain was divided into two parts, one with advection and diffusion, and the other with diffusion only (no macroscopic flow). Particles were exchanged between these domains at their common boundary by a direct process to process communication. By analysing weak and strong scaling, it was shown that the linear scaling characteristics of the lattice-Boltzmann solvers were not compromised by their coupling.
Sähkömagneettisen sironnan numeerinen simulointi
High-Reynolds-number turbulent cavity flow using the lattice Boltzmann method
We present a boundary condition scheme for the lattice Boltzmann method that has significantly improved stability for modeling turbulent flows while maintaining excellent parallel scalability. Simulations of a three-dimensional lid-driven cavity flow are found to be stable up to the unprecedented Reynolds number $\mathrm{Re}=5\ifmmode\times\else\texttimes\fi{}{10}^{4}$ for this setup. Excellent agreement with energy balance equations, computational and experimental results are shown. We quantify rises in the production of turbulence and turbulent drag, and determine peak locations of turbulent production.
High-order regularization in lattice-Boltzmann equations
A lattice-Boltzmann equation (LBE) is the discrete counterpart of a continuous kinetic model. It can be derived using a Hermite polynomial expansion for the velocity distribution function. Since LBEs are characterized by discrete, finite representations of the microscopic velocity space, the expansion must be truncated and the appropriate order of truncation depends on the hydrodynamic problem under investigation. Here we consider a particular truncation where the non-equilibrium distribution is expanded on a par with the equilibrium distribution, except that the diffusive parts of high-order nonequilibrium moments are filtered, i.e., only the corresponding advective parts are retained afte…
Diffusion through thin membranes: Modeling across scales
From macroscopic to microscopic scales it is demonstrated that diffusion through membranes can be modeled using specific boundary conditions across them. The membranes are here considered thin in comparison to the overall size of the system. In a macroscopic scale the membrane is introduced as a transmission boundary condition, which enables an effective modeling of systems that involve multiple scales. In a mesoscopic scale, a numerical lattice-Boltzmann scheme with a partial-bounceback condition at the membrane is proposed and analyzed. It is shown that this mesoscopic approach provides a consistent approximation of the transmission boundary condition. Furthermore, analysis of the mesosco…
Investigation of an entropic stabilizer for the lattice-Boltzmann method
The lattice-Boltzmann (LB) method is commonly used for the simulation of fluid flows at the hydrodynamic level of description. Due to its kinetic theory origins, the standard LB schemes carry more degrees of freedom than strictly needed, e.g., for the approximation of solutions to the Navier-stokes equation. In particular, there is freedom in the details of the so-called collision operator. This aspect was recently utilized when an entropic stabilizer, based on the principle of maximizing local entropy, was proposed for the LB method [I. V. Karlin, F. Bosch, and S. S. Chikatamarla, ¨ Phys. Rev. E 90, 031302(R) (2014)]. The proposed stabilizer can be considered as an add-on or extension to b…
An efficient swap algorithm for the lattice Boltzmann method
During the last decade, the lattice-Boltzmann method (LBM) as a valuable tool in computational fluid dynamics has been increasingly acknowledged. The widespread application of LBM is partly due to the simplicity of its coding. The most well-known algorithms for the implementation of the standard lattice-Boltzmann equation (LBE) are the two-lattice and two-step algorithms. However, implementations of the two-lattice or the two-step algorithm suffer from high memory consumption or poor computational performance, respectively. Ultimately, the computing resources available decide which of the two disadvantages is more critical. Here we introduce a new algorithm, called the swap algorithm, for t…
Designing a graphics processing unit accelerated petaflop capable lattice Boltzmann solver: Read aligned data layouts and asynchronous communication
The lattice Boltzmann method is a well-established numerical approach for complex fluid flow simulations. Recently, general-purpose graphics processing units (GPUs) have become available as high-performance computing resources at large scale. We report on designing and implementing a lattice Boltzmann solver for multi-GPU systems that achieves 1.79 PFLOPS performance on 16,384 GPUs. To achieve this performance, we introduce a GPU compatible version of the so-called bundle data layout and eliminate the halo sites in order to improve data access alignment. Furthermore, we make use of the possibility to overlap data transfer between the host central processing unit and the device GPU with com…
Fluid flow simulations meet high-speed video : Computer vision comparison of droplet dynamics
Hypothesis While multiphase flows, particularly droplet dynamics, are ordinary in nature as well as in industrial processes, their mathematical and computational modelling continue to pose challenging research tasks - patent approaches for tackling them are yet to be found. The lack of analytical flow field solutions for non-trivial droplet dynamics hinders validation of computer simulations and, hence, their application in research problems. High-speed videos and computer vision algorithms can provide a viable approach to validate simulations directly against experiments. Experiments Droplets of water (or glycerol-water mixtures) impacting on both hydrophobic and superhydrophobic surfaces …
Transport properties of heterogeneous materials. Combining computerised X-ray micro-tomography and direct numerical simulations
Feasibility of a method for finding flow permeability of porous materials, based on combining computerised X-ray micro-tomography and numerical simulations, is assessed. The permeability is found by solving fluid flow through the complex 3D pore structures obtained by tomography for actual material samples. We estimate overall accuracy of the method and compare numerical and experimental results. Factors contributing to uncertainty of the method include numerical error arising from the finite resolution of tomographic images and the rather small sample size available with the present tomographic techniques. The total uncertainty of computed values of permeability is, however, not essentiall…