For example, all of these variants of the basic Lennard-Jones pair -style exist in LAMMPS: +
For example, all of these styles are variants of the basic +Lennard-Jones pair style pair_style lj/cut:
-Assuming you have built LAMMPS with the appropriate package, these styles can be invoked by specifying them explicitly in your input @@ -161,11 +162,17 @@ script. Or you can use the -suffix comma switch to invoke the accelerated versions automatically, without changing your input script. The suffix command allows you to set a suffix explicitly and -to turn off/on the comand-line switch setting, both from within your -input script. +to turn off and back on the comand-line switch setting, both from +within your input script.
-Styles with an "opt" suffix are part of the OPT package and typically -speed-up the pairwise calculations of your simulation by 5-25%. +
Styles with a "cuda" or "gpu" suffix are part of the USER-CUDA or GPU +packages, and can be run on NVIDIA GPUs associated with your CPUs. +The speed-up due to GPU usage depends on a variety of factors, as +discussed below. +
+Styles with a "kk" suffix are part of the KOKKOS package, and can be +run using OpenMP, pthreads, or on an NVIDIA GPU. The speed-up depends +on a variety of factors, as discussed below.
Styles with an "omp" suffix are part of the USER-OMP package and allow a pair-style to be run in multi-threaded mode using OpenMP. This can @@ -174,26 +181,26 @@ than cores is advantageous, e.g. when running with PPPM so that FFTs are run on fewer MPI processors or when the many MPI tasks would overload the available bandwidth for communication.
-Styles with a "gpu" or "cuda" suffix are part of the GPU or USER-CUDA -packages, and can be run on NVIDIA GPUs associated with your CPUs. -The speed-up due to GPU usage depends on a variety of factors, as -discussed below. +
Styles with an "opt" suffix are part of the OPT package and typically +speed-up the pairwise calculations of your simulation by 5-25%.
To see what styles are currently available in each of the accelerated packages, see Section_commands 5 of the manual. A list of accelerated styles is included in the pair, fix, -compute, and kspace sections. +compute, and kspace sections. The doc page for each indvidual style +(e.g. pair lj/cut or fix nve) will also +list any accelerated variants available for that style.
The following sections explain:
The final section compares and contrasts the GPU and USER-CUDA -packages, since they are both designed to use NVIDIA GPU hardware. +packages, since they are both designed to use NVIDIA hardware.
make yes-opt make machine-
If your input script uses one of the OPT pair styles, -you can run it as follows: +
If your input script uses one of the OPT pair styles, you can run it +as follows:
lmp_machine -sf opt < in.script mpirun -np 4 lmp_machine -sf opt < in.script @@ -226,12 +233,13 @@ to 20% savings.5.5 USER-OMP package
-The USER-OMP package was developed by Axel Kohlmeyer at Temple University. -It provides multi-threaded versions of most pair styles, all dihedral -styles and a few fixes in LAMMPS. The package currently uses the OpenMP -interface which requires using a specific compiler flag in the makefile -to enable multiple threads; without this flag the corresponding pair -styles will still be compiled and work, but do not support multi-threading. +
The USER-OMP package was developed by Axel Kohlmeyer at Temple +University. It provides multi-threaded versions of most pair styles, +all dihedral styles, and a few fixes in LAMMPS. The package currently +uses the OpenMP interface which requires using a specific compiler +flag in the makefile to enable multiple threads; without this flag the +corresponding pair styles will still be compiled and work, but do not +support multi-threading.
Building LAMMPS with the USER-OMP package:
@@ -264,18 +272,19 @@ env OMP_NUM_THREADS=2 mpirun -np 2 lmp_machine -sf omp -in in.script mpirun -x OMP_NUM_THREADS=2 -np 2 lmp_machine -sf omp -in in.script
The value of the environment variable OMP_NUM_THREADS determines how -many threads per MPI task are launched. All three examples above use -a total of 4 CPU cores. For different MPI implementations the method -to pass the OMP_NUM_THREADS environment variable to all processes is -different. Two different variants, one for MPICH and OpenMPI, respectively -are shown above. Please check the documentation of your MPI installation -for additional details. Alternatively, the value provided by OMP_NUM_THREADS -can be overridded with the package omp command. -Depending on which styles are accelerated in your input, you should -see a reduction in the "Pair time" and/or "Bond time" and "Loop time" -printed out at the end of the run. The optimal ratio of MPI to OpenMP -can vary a lot and should always be confirmed through some benchmark -runs for the current system and on the current machine. +many threads per MPI task are launched. All three examples above use a +total of 4 CPU cores. For different MPI implementations the method to +pass the OMP_NUM_THREADS environment variable to all processes is +different. Two different variants, one for MPICH and OpenMPI, +respectively are shown above. Please check the documentation of your +MPI installation for additional details. Alternatively, the value +provided by OMP_NUM_THREADS can be overridded with the package +omp command. Depending on which styles are accelerated +in your input, you should see a reduction in the "Pair time" and/or +"Bond time" and "Loop time" printed out at the end of the run. The +optimal ratio of MPI to OpenMP can vary a lot and should always be +confirmed through some benchmark runs for the current system and on +the current machine.
Restrictions:
@@ -293,53 +302,55 @@ On the other hand, in many cases you still want to use the omp version all contain optimizations similar to those in the OPT package, which can result in serial speedup. -Using multi-threading is most effective under the following circumstances: +
Using multi-threading is most effective under the following +circumstances:
-The best parallel efficiency from omp styles is typically -achieved when there is at least one MPI task per physical -processor, i.e. socket or die. +
The best parallel efficiency from omp styles is typically achieved +when there is at least one MPI task per physical processor, +i.e. socket or die.
Using threads on hyper-threading enabled cores is usually counterproductive, as the cost in additional memory bandwidth -requirements is not offset by the gain in CPU utilization -through hyper-threading. +requirements is not offset by the gain in CPU utilization through +hyper-threading.
A description of the multi-threading strategy and some performance -examples are presented here +examples are presented +here
NOTE: - discuss 3 precisions - if change, also have to re-link with LAMMPS - always use newton off - expt with differing numbers of CPUs vs GPU - can't tell what is fastest - give command line switches in examples -
-I am not very clear to the meaning of "Max Mem / Proc" -in the "GPU Time Info (average)". -Is it the maximal of GPU memory used by one CPU core? -
-It is the maximum memory used at one time on the GPU for data storage by -a single MPI process. - Mike -
Hardware and software requirements:
-To use this package, you currently need to have specific NVIDIA -hardware and install specific NVIDIA CUDA software on your system: +
To use this package, you currently need to have an NVIDIA GPU and +install the NVIDIA Cuda software on your system:
-Before building the library, you can set the precision it will use by +editing the CUDA_PREC setting in the Makefile you are using, as +follows: +
+CUDA_PREC = -D_SINGLE_SINGLE # Single precision for all calculations +CUDA_PREC = -D_DOUBLE_DOUBLE # Double precision for all calculations +CUDA_PREC = -D_SINGLE_DOUBLE # Accumulation of forces, etc, in double ++
The last setting is the mixed mode referred to above. Note that your +GPU must support double precision to use either the 2nd or 3rd of +these settings. +
+To build the library, then type:
cd lammps/lib/gpu make -f Makefile.linux @@ -424,41 +439,60 @@ set appropriately to include the paths and settings for the CUDA system software on your machine. See src/MAKE/Makefile.g++ for an example. -GPU configuration +
Also note that if you change the GPU library precision, you need to +re-build the entire library. You should do a "clean" first, +e.g. "make -f Makefile.linux clean". Then you must also re-build +LAMMPS if the library precision has changed, so that it re-links with +the new library.
-When using GPUs, you are restricted to one physical GPU per LAMMPS -process, which is an MPI process running on a single core or -processor. Multiple MPI processes (CPU cores) can share a single GPU, -and in many cases it will be more efficient to run this way. +
Running an input script:
-Input script requirements: +
The examples/gpu and bench/GPU directories have scripts that can be +run with the GPU package, as well as detailed instructions on how to +run them.
-Additional input script requirements to run pair or PPPM styles with a +
The total number of MPI tasks used by LAMMPS (one or multiple per +compute node) is set in the usual manner via the mpirun or mpiexec +commands, and is independent of the GPU package. +
+When using the GPU package, you cannot assign more than one physical +GPU to an MPI task. However multiple MPI tasks can share the same +GPU, and in many cases it will be more efficient to run this way. +
+Input script requirements to run using pair or PPPM styles with a gpu suffix are as follows:
-
As an example, if you have two GPUs per node and 8 CPU cores per node, -and would like to run on 4 nodes (32 cores) with dynamic balancing of -force calculation across CPU and GPU cores, you could specify +
The default for the package gpu command is to have all +the MPI tasks on the compute node use a single GPU. If you have +multiple GPUs per node, then be sure to create one or more MPI tasks +per GPU, and use the first/last settings in the package +gpu command to include all the GPU IDs on the node. +E.g. first = 0, last = 1, for 2 GPUs. For example, on an 8-core 2-GPU +compute node, if you assign 8 MPI tasks to the node, the following +command in the input script
-package gpu force/neigh 0 1 -1 --
In this case, all CPU cores and GPU devices on the nodes would be -utilized. Each GPU device would be shared by 4 CPU cores. The CPU -cores would perform force calculations for some fraction of the -particles at the same time the GPUs performed force calculation for -the other particles. +
package gpu force/neigh 0 1 -1 +
+would speciy each GPU is shared by 4 MPI tasks. The final -1 will +dynamically balance force calculations across the CPU cores and GPUs. +I.e. each CPU core will perform force calculations for some small +fraction of the particles, at the same time the GPUs perform force +calcaultions for the majority of the particles.
Timing output:
@@ -482,19 +516,30 @@ screen output (not in the log file) at the end of each run. These timings represent total time spent on the GPU for each routine, regardless of asynchronous CPU calculations. +The output section "GPU Time Info (average)" reports "Max Mem / Proc". +This is the maximum memory used at one time on the GPU for data +storage by a single MPI process. +
Performance tips:
-Generally speaking, for best performance, you should use multiple CPUs -per GPU, as provided my most multi-core CPU/GPU configurations. +
You should experiment with how many MPI tasks per GPU to use to see +what gives the best performance for your problem. This is a function +of your problem size and what pair style you are using. Likewise, you +should also experiment with the precision setting for the GPU library +to see if single or mixed precision will give accurate results, since +they will typically be faster.
-Because of the large number of cores within each GPU device, it may be -more efficient to run on fewer processes per GPU when the number of -particles per MPI process is small (100's of particles); this can be -necessary to keep the GPU cores busy. +
Using multiple MPI tasks per GPU will often give the best performance, +as allowed my most multi-core CPU/GPU configurations.
-See the lammps/lib/gpu/README file for instructions on how to build -the GPU library for single, mixed, or double precision. The latter -requires that your GPU card support double precision. +
If the number of particles per MPI task is small (e.g. 100s of +particles), it can be more eefficient to run with fewer MPI tasks per +GPU, even if you do not use all the cores on the compute node. +
+The Benchmark page of the LAMMPS +web site gives GPU performance on a desktop machine and the Titan HPC +platform at ORNL for several of the LAMMPS benchmarks, as a function +of problem size and number of compute nodes.
The KOKKOS package contains versions of pair, fix, and atom styles +that use data structures and methods and macros provided by the Kokkos +library, which is included with LAMMPS in lib/kokkos. +
+Kokkos is a C++ library +that provides two key abstractions for an application like LAMMPS. +First, it allows a single implementation of an application kernel +(e.g. a pair style) to run efficiently on different kinds of hardware +(GPU, Intel Phi, many-core chip). +
+Second, it provides data abstractions to adjust (at compile time) the +memory layout of basic data structures like 2d and 3d arrays and allow +the transparent utilization of special hardware load and store units. +Such data structures are used in LAMMPS to store atom coordinates or +forces or neighbor lists. The layout is chosen to optimize +performance on different platforms. Again this operation is hidden +from the developer, and does not affect how the single implementation +of the kernel is coded. +
+These abstractions are set at build time, when LAMMPS is compiled with +the KOKKOS package installed. This is done by selecting a "host" and +"device" to build for, compatible with the compute nodes in your +machine. Note that if you are running on a desktop machine, you +typically have one compute node. On a cluster or supercomputer there +may be dozens or 1000s of compute nodes. The procedure for building +and running with the Kokkos library is the same, no matter how many +nodes you run on. +
+All Kokkos operations occur within the context of an individual MPI +task running on a single node of the machine. The total number of MPI +tasks used by LAMMPS (one or multiple per compute node) is set in the +usual manner via the mpirun or mpiexec commands, and is independent of +Kokkos. +
+Kokkos provides support for one or two modes of execution per MPI +task. This means that some computational tasks (pairwise +interactions, neighbor list builds, time integration, etc) are +parallelized in one or the other of the two modes. The first mode is +called the "host" and is one or more threads running on one or more +physical CPUs (within the node). Currently, both multi-core CPUs and +an Intel Phi processor (running in native mode) are supported. The +second mode is called the "device" and is an accelerator chip of some +kind. Currently only an NVIDIA GPU is supported. If your compute +node does not have a GPU, then there is only one mode of execution, +i.e. the host and device are the same. +
+IMPORTNANT NOTE: Currently, if using GPUs, you should set the number +of MPI tasks per compute node to be equal to the number of GPUs per +compute node. In the future Kokkos will support assigning one GPU to +multiple MPI tasks or using multiple GPUs per MPI task. Currently +Kokkos does not support AMD GPUs due to limits in the available +backend programming models (in particular relative extensive C++ +support is required for the Kernel language). This is expected to +change in the future. +
+Here are several examples of how to build LAMMPS and run a simulation +using the KOKKOS package for typical compute node configurations. +Note that the -np setting for the mpirun command in these examples are +for a run on a single node. To scale these examples up to run on a +system with N compute nodes, simply multiply the -np setting by N. +
+All the build steps are performed from within the src directory. All +the run steps are performed in the bench directory using the in.lj +input script. It is assumed the LAMMPS executable has been copied to +that directory or whatever directory the runs are being performed in. +Details of the various options are discussed below. +
+Compute node(s) = dual hex-core CPUs and no GPU: +
+make yes-kokkos # install the KOKKOS package +make g++ OMP=yes # build with OpenMP, no CUDA ++
mpirun -np 12 lmp_g++ -k off < in.lj # MPI-only mode with no Kokkos +mpirun -np 12 lmp_g++ -sf kk < in.lj # MPI-only mode with Kokkos +mpirun -np 1 lmp_g++ -k on t 12 -sf kk < in.lj # one MPI task, 12 threads +mpirun -np 2 lmp_g++ -k on t 6 -sf kk < in.lj # two MPI tasks, 6 threads/task ++
Compute node(s) = Intel Phi with 61 cores: +
+make yes-kokkos +make g++ OMP=yes MIC=yes # build with OpenMP for Phi ++
mpirun -np 12 lmp_g++ -k on t 20 -sf kk < in.lj # 12*20 = 240 total cores +mpirun -np 15 lmp_g++ -k on t 16 -sf kk < in.lj +mpirun -np 30 lmp_g++ -k on t 8 -sf kk < in.lj +mpirun -np 1 lmp_g++ -k on t 240 -sf kk < in.lj ++
Compute node(s) = dual hex-core CPUs and a single GPU: +
+make yes-kokkos +make cuda CUDA=yes # build for GPU, use src/MAKE/Makefile.cuda ++
mpirun -np 1 lmp_cuda -k on t 6 -sf kk < in.lj ++
Compute node(s) = dual 8-core CPUs and 2 GPUs: +
+make yes-kokkos +make cuda CUDA=yes ++
mpirun -np 2 lmp_cuda -k on t 8 g 2 -sf kk < in.lj # use both GPUs, one per MPI task ++
Building LAMMPS with the KOKKOS package: +
+A summary of the build process is given here. More details and all +the available make variable options are given in this +section of the manual. +
+From the src directory, type +
+make yes-kokkos ++
to include the KOKKOS package. Then perform a normal LAMMPS build, +with additional make variable specifications to choose the host and +device you will run the resulting executable on, e.g. +
+make g++ OMP=yes +make cuda CUDA=yes ++
As illustrated above, the most important variables to set are OMP, +CUDA, and MIC. The default settings are OMP=yes, CUDA=no, MIC=no +Setting OMP to yes will use OpenMP for threading on the host, as +well as on the device (if no GPU is present). Setting CUDA to yes +will use one or more GPUs as the device. Setting MIC=yes is necessary +when building for an Intel Phi processor. +
+Note that to use a GPU, you must use a lo-level Makefile, +e.g. src/MAKE/Makefile.cuda as included in the LAMMPS distro, which +uses the NVIDA "nvcc" compiler. You must check that the CCFLAGS -arch +setting is appropriate for your NVIDIA hardware and installed +software. Typical values for -arch are given in this +section of the manual, as well as other +settings that must be included in the lo-level Makefile, if you create +your own. +
+Input scripts and use of command-line switches -kokkos and -suffix: +
+To use any Kokkos-enabled style provided in the KOKKOS package, you +must use a Kokkos-enabled atom style. LAMMPS will give an error if +you do not do this. +
+There are two command-line switches relevant to using Kokkos, -k or +-kokkos, and -sf or -suffix. They are described in detail in this +section of the manual. +
+Here are common options to use: +
+Use of package command options: +
+Using the package kokkos command in an input script +allows choice of options for neighbor lists and communication. See +the package command doc page for details and default +settings. +
+Experimenting with different styles of neighbor lists or inter-node +communication can provide a speed-up for specific calculations. +
+Running on a multi-core CPU: +
+Build with OMP=yes (the default) and CUDA=no (the default). +
+If N is the number of physical cores/node, then the number of MPI +tasks/node * number of threads/task should not exceed N, and should +typically equal N. Note that the default threads/task is 1, as set by +the "t" keyword of the -k command-line +switch. If you do not change this, no +additional parallelism (beyond MPI) will be invoked on the host +CPU(s). +
+You can compare the performance running in different modes: +
+Examples of mpirun commands in these modes, for nodes with dual +hex-core CPUs and no GPU, are shown above. +
+Running on GPUs: +
+Build with CUDA=yes, using src/MAKE/Makefile.cuda. Insure the setting +for CUDA_PATH in lib/kokkos/Makefile.lammps is correct for your Cuda +software installation. Insure the -arch setting in +src/MAKE/Makefile.cuda is correct for your GPU hardware/software (see +this section of the manual for details. +
+The -np setting of the mpirun command should set the number of MPI +tasks/node to be equal to the # of physical GPUs on the node. +
+Use the -kokkos command-line switch to +specify the number of GPUs per node, and the number of threads per MPI +task. As above for multi-core CPUs (and no GPU), if N is the number +of physical cores/node, then the number of MPI tasks/node * number of +threads/task should not exceed N. With one GPU (and one MPI task) it +may be faster to use less than all the available cores, by setting +threads/task to a smaller value. This is because using all the cores +on a dual-socket node will incur extra cost to copy memory from the +2nd socket to the GPU. +
+Examples of mpirun commands that follow these rules, for nodes with +dual hex-core CPUs and one or two GPUs, are shown above. +
+Running on an Intel Phi: +
+Kokkos only uses Intel Phi processors in their "native" mode, i.e. +not hosted by a CPU. +
+Build with OMP=yes (the default) and MIC=yes. The latter +insures code is correctly compiled for the Intel Phi. The +OMP setting means OpenMP will be used for parallelization +on the Phi, which is currently the best option within +Kokkos. In the future, other options may be added. +
+Current-generation Intel Phi chips have either 61 or 57 cores. One +core should be excluded to run the OS, leaving 60 or 56 cores. Each +core is hyperthreaded, so there are effectively N = 240 (4*60) or N = +224 (4*56) cores to run on. +
+The -np setting of the mpirun command sets the number of MPI +tasks/node. The "-k on t Nt" command-line switch sets the number of +threads/task as Nt. The product of these 2 values should be N, i.e. +240 or 224. Also, the number of threads/task should be a multiple of +4 so that logical threads from more than one MPI task do not run on +the same physical core. +
+Examples of mpirun commands that follow these rules, for Intel Phi +nodes with 61 cores, are shown above. +
+Examples and benchmarks: +
+The examples/kokkos and bench/KOKKOS directories have scripts that can +be run with the KOKKOS package, as well as detailed instructions on +how to run them. +
+IMPORTANT NOTE: the bench/KOKKOS directory does not yet exist. It +will be added later. +
+Additional performance issues: +
+When using threads (OpenMP or pthreads), it is important for +performance to bind the threads to physical cores, so they do not +migrate during a simulation. The same is true for MPI tasks, but the +default binding rules implemented for various MPI versions, do not +account for thread binding. +
+Thus if you use more than one thread per MPI task, you should insure +MPI tasks are bound to CPU sockets. Furthermore, use thread affinity +environment variables from the OpenMP runtime when using OpenMP and +compile with hwloc support when using pthreads. With OpenMP 3.1 (gcc +4.7 or later, intel 12 or later) setting the environment variable +OMP_PROC_BIND=true should be sufficient. A typical mpirun command +should set these flags: +
+OpenMPI 1.8: mpirun -np 2 -bind-to socket -map-by socket ./lmp_openmpi ... +Mvapich2 2.0: mpiexec -np 2 -bind-to socket -map-by socket ./lmp_mvapich ... ++
When using a GPU, you will achieve the best performance if your input +script does not use any fix or compute styles which are not yet +Kokkos-enabled. This allows data to stay on the GPU for multiple +timesteps, without being copied back to the host CPU. Invoking a +non-Kokkos fix or compute, or performing I/O for +thermo or dump output will cause data +to be copied back to the CPU. +
+You cannot yet assign multiple MPI tasks to the same GPU with the +KOKKOS package. We plan to support this in the future, similar to the +GPU package in LAMMPS. +
+You cannot yet use both the host (multi-threaded) and device (GPU) +together to compute pairwise interactions with the KOKKOS package. We +hope to support this in the future, similar to the GPU package in +LAMMPS. +
Both the GPU and USER-CUDA packages accelerate a LAMMPS calculation using NVIDIA hardware, but they do it in different ways. diff --git a/doc/Section_accelerate.txt b/doc/Section_accelerate.txt index 1e995d5c72..a4252cf177 100644 --- a/doc/Section_accelerate.txt +++ b/doc/Section_accelerate.txt @@ -21,7 +21,8 @@ kinds of machines. 5.5 "USER-OMP package"_#acc_5 5.6 "GPU package"_#acc_6 5.7 "USER-CUDA package"_#acc_7 -5.8 "Comparison of GPU and USER-CUDA packages"_#acc_8 :all(b) +5.8 "KOKKOS package"_#acc_8 +5.9 "Comparison of GPU and USER-CUDA packages"_#acc_9 :all(b) :line :line @@ -142,14 +143,14 @@ command is identical, their functionality is the same, and the numerical results it produces should also be identical, except for precision and round-off issues. -For example, all of these variants of the basic Lennard-Jones pair -style exist in LAMMPS: +For example, all of these styles are variants of the basic +Lennard-Jones pair style "pair_style lj/cut"_pair_lj.html: -"pair_style lj/cut"_pair_lj.html -"pair_style lj/cut/opt"_pair_lj.html -"pair_style lj/cut/omp"_pair_lj.html +"pair_style lj/cut/cuda"_pair_lj.html "pair_style lj/cut/gpu"_pair_lj.html -"pair_style lj/cut/cuda"_pair_lj.html :ul +"pair_style lj/cut/kk"_pair_lj.html +"pair_style lj/cut/omp"_pair_lj.html +"pair_style lj/cut/opt"_pair_lj.html :ul Assuming you have built LAMMPS with the appropriate package, these styles can be invoked by specifying them explicitly in your input @@ -157,11 +158,17 @@ script. Or you can use the "-suffix command-line switch"_Section_start.html#start_7 to invoke the accelerated versions automatically, without changing your input script. The "suffix"_suffix.html command allows you to set a suffix explicitly and -to turn off/on the comand-line switch setting, both from within your -input script. +to turn off and back on the comand-line switch setting, both from +within your input script. -Styles with an "opt" suffix are part of the OPT package and typically -speed-up the pairwise calculations of your simulation by 5-25%. +Styles with a "cuda" or "gpu" suffix are part of the USER-CUDA or GPU +packages, and can be run on NVIDIA GPUs associated with your CPUs. +The speed-up due to GPU usage depends on a variety of factors, as +discussed below. + +Styles with a "kk" suffix are part of the KOKKOS package, and can be +run using OpenMP, pthreads, or on an NVIDIA GPU. The speed-up depends +on a variety of factors, as discussed below. Styles with an "omp" suffix are part of the USER-OMP package and allow a pair-style to be run in multi-threaded mode using OpenMP. This can @@ -170,26 +177,26 @@ than cores is advantageous, e.g. when running with PPPM so that FFTs are run on fewer MPI processors or when the many MPI tasks would overload the available bandwidth for communication. -Styles with a "gpu" or "cuda" suffix are part of the GPU or USER-CUDA -packages, and can be run on NVIDIA GPUs associated with your CPUs. -The speed-up due to GPU usage depends on a variety of factors, as -discussed below. +Styles with an "opt" suffix are part of the OPT package and typically +speed-up the pairwise calculations of your simulation by 5-25%. To see what styles are currently available in each of the accelerated packages, see "Section_commands 5"_Section_commands.html#cmd_5 of the manual. A list of accelerated styles is included in the pair, fix, -compute, and kspace sections. +compute, and kspace sections. The doc page for each indvidual style +(e.g. "pair lj/cut"_pair_lj.html or "fix nve"_fix_nve.html) will also +list any accelerated variants available for that style. The following sections explain: what hardware and software the accelerated styles require -how to build LAMMPS with the accelerated packages in place +how to build LAMMPS with the accelerated package in place what changes (if any) are needed in your input scripts guidelines for best performance speed-ups you can expect :ul The final section compares and contrasts the GPU and USER-CUDA -packages, since they are both designed to use NVIDIA GPU hardware. +packages, since they are both designed to use NVIDIA hardware. :line @@ -208,8 +215,8 @@ dependencies: make yes-opt make machine :pre -If your input script uses one of the OPT pair styles, -you can run it as follows: +If your input script uses one of the OPT pair styles, you can run it +as follows: lmp_machine -sf opt -in in.script mpirun -np 4 lmp_machine -sf opt -in in.script :pre @@ -222,12 +229,13 @@ to 20% savings. 5.5 USER-OMP package :h4,link(acc_5) -The USER-OMP package was developed by Axel Kohlmeyer at Temple University. -It provides multi-threaded versions of most pair styles, all dihedral -styles and a few fixes in LAMMPS. The package currently uses the OpenMP -interface which requires using a specific compiler flag in the makefile -to enable multiple threads; without this flag the corresponding pair -styles will still be compiled and work, but do not support multi-threading. +The USER-OMP package was developed by Axel Kohlmeyer at Temple +University. It provides multi-threaded versions of most pair styles, +all dihedral styles, and a few fixes in LAMMPS. The package currently +uses the OpenMP interface which requires using a specific compiler +flag in the makefile to enable multiple threads; without this flag the +corresponding pair styles will still be compiled and work, but do not +support multi-threading. [Building LAMMPS with the USER-OMP package:] @@ -260,18 +268,19 @@ env OMP_NUM_THREADS=2 mpirun -np 2 lmp_machine -sf omp -in in.script mpirun -x OMP_NUM_THREADS=2 -np 2 lmp_machine -sf omp -in in.script :pre The value of the environment variable OMP_NUM_THREADS determines how -many threads per MPI task are launched. All three examples above use -a total of 4 CPU cores. For different MPI implementations the method -to pass the OMP_NUM_THREADS environment variable to all processes is -different. Two different variants, one for MPICH and OpenMPI, respectively -are shown above. Please check the documentation of your MPI installation -for additional details. Alternatively, the value provided by OMP_NUM_THREADS -can be overridded with the "package omp"_package.html command. -Depending on which styles are accelerated in your input, you should -see a reduction in the "Pair time" and/or "Bond time" and "Loop time" -printed out at the end of the run. The optimal ratio of MPI to OpenMP -can vary a lot and should always be confirmed through some benchmark -runs for the current system and on the current machine. +many threads per MPI task are launched. All three examples above use a +total of 4 CPU cores. For different MPI implementations the method to +pass the OMP_NUM_THREADS environment variable to all processes is +different. Two different variants, one for MPICH and OpenMPI, +respectively are shown above. Please check the documentation of your +MPI installation for additional details. Alternatively, the value +provided by OMP_NUM_THREADS can be overridded with the "package +omp"_package.html command. Depending on which styles are accelerated +in your input, you should see a reduction in the "Pair time" and/or +"Bond time" and "Loop time" printed out at the end of the run. The +optimal ratio of MPI to OpenMP can vary a lot and should always be +confirmed through some benchmark runs for the current system and on +the current machine. [Restrictions:] @@ -292,53 +301,55 @@ On the other hand, in many cases you still want to use the {omp} version all contain optimizations similar to those in the OPT package, which can result in serial speedup. -Using multi-threading is most effective under the following circumstances: +Using multi-threading is most effective under the following +circumstances: -Individual compute nodes have a significant number of CPU cores -but the CPU itself has limited memory bandwidth, e.g. Intel Xeon 53xx -(Clovertown) and 54xx (Harpertown) quad core processors. Running -one MPI task per CPU core will result in significant performance -degradation, so that running with 4 or even only 2 MPI tasks per -nodes is faster. Running in hybrid MPI+OpenMP mode will reduce the -inter-node communication bandwidth contention in the same way, -but offers and additional speedup from utilizing the otherwise -idle CPU cores. :ulb,l +Individual compute nodes have a significant number of CPU cores but +the CPU itself has limited memory bandwidth, e.g. Intel Xeon 53xx +(Clovertown) and 54xx (Harpertown) quad core processors. Running one +MPI task per CPU core will result in significant performance +degradation, so that running with 4 or even only 2 MPI tasks per nodes +is faster. Running in hybrid MPI+OpenMP mode will reduce the +inter-node communication bandwidth contention in the same way, but +offers and additional speedup from utilizing the otherwise idle CPU +cores. :ulb,l The interconnect used for MPI communication is not able to provide -sufficient bandwidth for a large number of MPI tasks per node. -This applies for example to running over gigabit ethernet or -on Cray XT4 or XT5 series supercomputers. Same as in the aforementioned -case this effect worsens with using an increasing number of nodes. :l +sufficient bandwidth for a large number of MPI tasks per node. This +applies for example to running over gigabit ethernet or on Cray XT4 or +XT5 series supercomputers. Same as in the aforementioned case this +effect worsens with using an increasing number of nodes. :l -The input is a system that has an inhomogeneous particle density -which cannot be mapped well to the domain decomposition scheme -that LAMMPS employs. While this can be to some degree alleviated -through using the "processors"_processors.html keyword, multi-threading -provides a parallelism that parallelizes over the number of particles -not their distribution in space. :l +The input is a system that has an inhomogeneous particle density which +cannot be mapped well to the domain decomposition scheme that LAMMPS +employs. While this can be to some degree alleviated through using the +"processors"_processors.html keyword, multi-threading provides a +parallelism that parallelizes over the number of particles not their +distribution in space. :l Finally, multi-threaded styles can improve performance when running LAMMPS in "capability mode", i.e. near the point where the MPI -parallelism scales out. This can happen in particular when using -as kspace style for long-range electrostatics. Here the scaling -of the kspace style is the performance limiting factor and using -multi-threaded styles allows to operate the kspace style at the -limit of scaling and then increase performance parallelizing -the real space calculations with hybrid MPI+OpenMP. Sometimes -additional speedup can be achived by increasing the real-space -coulomb cutoff and thus reducing the work in the kspace part. :l,ule +parallelism scales out. This can happen in particular when using as +kspace style for long-range electrostatics. Here the scaling of the +kspace style is the performance limiting factor and using +multi-threaded styles allows to operate the kspace style at the limit +of scaling and then increase performance parallelizing the real space +calculations with hybrid MPI+OpenMP. Sometimes additional speedup can +be achived by increasing the real-space coulomb cutoff and thus +reducing the work in the kspace part. :l,ule -The best parallel efficiency from {omp} styles is typically -achieved when there is at least one MPI task per physical -processor, i.e. socket or die. +The best parallel efficiency from {omp} styles is typically achieved +when there is at least one MPI task per physical processor, +i.e. socket or die. Using threads on hyper-threading enabled cores is usually counterproductive, as the cost in additional memory bandwidth -requirements is not offset by the gain in CPU utilization -through hyper-threading. +requirements is not offset by the gain in CPU utilization through +hyper-threading. A description of the multi-threading strategy and some performance -examples are "presented here"_http://sites.google.com/site/akohlmey/software/lammps-icms/lammps-icms-tms2011-talk.pdf?attredirects=0&d=1 +examples are "presented +here"_http://sites.google.com/site/akohlmey/software/lammps-icms/lammps-icms-tms2011-talk.pdf?attredirects=0&d=1 :line @@ -365,36 +376,23 @@ between processors, runs on the CPU. :l Asynchronous force computations can be performed simultaneously on the CPU(s) and GPU. :l +It allows for GPU computations to be performed in single or double +precision, or in mixed-mode precision. where pairwise forces are +cmoputed in single precision, but accumulated into double-precision +force vectors. :l + LAMMPS-specific code is in the GPU package. It makes calls to a generic GPU library in the lib/gpu directory. This library provides NVIDIA support as well as more general OpenCL support, so that the same functionality can eventually be supported on a variety of GPU hardware. :l,ule - - -NOTE: - discuss 3 precisions - if change, also have to re-link with LAMMPS - always use newton off - expt with differing numbers of CPUs vs GPU - can't tell what is fastest - give command line switches in examples - - -I am not very clear to the meaning of "Max Mem / Proc" -in the "GPU Time Info (average)". -Is it the maximal of GPU memory used by one CPU core? - -It is the maximum memory used at one time on the GPU for data storage by -a single MPI process. - Mike - - [Hardware and software requirements:] -To use this package, you currently need to have specific NVIDIA -hardware and install specific NVIDIA CUDA software on your system: +To use this package, you currently need to have an NVIDIA GPU and +install the NVIDIA Cuda software on your system: -Check if you have an NVIDIA card: cat /proc/driver/nvidia/cards/0 +Check if you have an NVIDIA GPU: cat /proc/driver/nvidia/cards/0 Go to http://www.nvidia.com/object/cuda_get.html Install a driver and toolkit appropriate for your system (SDK is not necessary) Follow the instructions in lammps/lib/gpu/README to build the library (see below) @@ -406,8 +404,21 @@ As with other packages that include a separately compiled library, you need to first build the GPU library, before building LAMMPS itself. General instructions for doing this are in "this section"_Section_start.html#start_3 of the manual. For this package, -do the following, using a Makefile in lib/gpu appropriate for your -system: +use a Makefile in lib/gpu appropriate for your system. + +Before building the library, you can set the precision it will use by +editing the CUDA_PREC setting in the Makefile you are using, as +follows: + +CUDA_PREC = -D_SINGLE_SINGLE # Single precision for all calculations +CUDA_PREC = -D_DOUBLE_DOUBLE # Double precision for all calculations +CUDA_PREC = -D_SINGLE_DOUBLE # Accumulation of forces, etc, in double :pre + +The last setting is the mixed mode referred to above. Note that your +GPU must support double precision to use either the 2nd or 3rd of +these settings. + +To build the library, then type: cd lammps/lib/gpu make -f Makefile.linux @@ -427,41 +438,60 @@ set appropriately to include the paths and settings for the CUDA system software on your machine. See src/MAKE/Makefile.g++ for an example. -[GPU configuration] +Also note that if you change the GPU library precision, you need to +re-build the entire library. You should do a "clean" first, +e.g. "make -f Makefile.linux clean". Then you must also re-build +LAMMPS if the library precision has changed, so that it re-links with +the new library. -When using GPUs, you are restricted to one physical GPU per LAMMPS -process, which is an MPI process running on a single core or -processor. Multiple MPI processes (CPU cores) can share a single GPU, -and in many cases it will be more efficient to run this way. +[Running an input script:] -[Input script requirements:] +The examples/gpu and bench/GPU directories have scripts that can be +run with the GPU package, as well as detailed instructions on how to +run them. -Additional input script requirements to run pair or PPPM styles with a +The total number of MPI tasks used by LAMMPS (one or multiple per +compute node) is set in the usual manner via the mpirun or mpiexec +commands, and is independent of the GPU package. + +When using the GPU package, you cannot assign more than one physical +GPU to an MPI task. However multiple MPI tasks can share the same +GPU, and in many cases it will be more efficient to run this way. + +Input script requirements to run using pair or PPPM styles with a {gpu} suffix are as follows: -To invoke specific styles from the GPU package, you can either append -"gpu" to the style name (e.g. pair_style lj/cut/gpu), or use the -"-suffix command-line switch"_Section_start.html#start_7, or use the -"suffix"_suffix.html command. :ulb,l +To invoke specific styles from the GPU package, either append "gpu" to +the style name (e.g. pair_style lj/cut/gpu), or use the "-suffix +command-line switch"_Section_start.html#start_7, or use the +"suffix"_suffix.html command in the input script. :ulb,l -The "newton pair"_newton.html setting must be {off}. :l +The "newton pair"_newton.html setting in the input script must be +{off}. :l -The "package gpu"_package.html command must be used near the beginning -of your script to control the GPU selection and initialization -settings. It also has an option to enable asynchronous splitting of -force computations between the CPUs and GPUs. :l,ule +Unless the "-suffix gpu command-line +switch"_Section_start.html#start_7 is used, the "package +gpu"_package.html command must be used near the beginning of the +script to control the GPU selection and initialization settings. It +also has an option to enable asynchronous splitting of force +computations between the CPUs and GPUs. :l,ule -As an example, if you have two GPUs per node and 8 CPU cores per node, -and would like to run on 4 nodes (32 cores) with dynamic balancing of -force calculation across CPU and GPU cores, you could specify +The default for the "package gpu"_package.html command is to have all +the MPI tasks on the compute node use a single GPU. If you have +multiple GPUs per node, then be sure to create one or more MPI tasks +per GPU, and use the first/last settings in the "package +gpu"_package.html command to include all the GPU IDs on the node. +E.g. first = 0, last = 1, for 2 GPUs. For example, on an 8-core 2-GPU +compute node, if you assign 8 MPI tasks to the node, the following +command in the input script -package gpu force/neigh 0 1 -1 :pre +package gpu force/neigh 0 1 -1 -In this case, all CPU cores and GPU devices on the nodes would be -utilized. Each GPU device would be shared by 4 CPU cores. The CPU -cores would perform force calculations for some fraction of the -particles at the same time the GPUs performed force calculation for -the other particles. +would speciy each GPU is shared by 4 MPI tasks. The final -1 will +dynamically balance force calculations across the CPU cores and GPUs. +I.e. each CPU core will perform force calculations for some small +fraction of the particles, at the same time the GPUs perform force +calcaultions for the majority of the particles. [Timing output:] @@ -485,19 +515,30 @@ screen output (not in the log file) at the end of each run. These timings represent total time spent on the GPU for each routine, regardless of asynchronous CPU calculations. +The output section "GPU Time Info (average)" reports "Max Mem / Proc". +This is the maximum memory used at one time on the GPU for data +storage by a single MPI process. + [Performance tips:] -Generally speaking, for best performance, you should use multiple CPUs -per GPU, as provided my most multi-core CPU/GPU configurations. +You should experiment with how many MPI tasks per GPU to use to see +what gives the best performance for your problem. This is a function +of your problem size and what pair style you are using. Likewise, you +should also experiment with the precision setting for the GPU library +to see if single or mixed precision will give accurate results, since +they will typically be faster. -Because of the large number of cores within each GPU device, it may be -more efficient to run on fewer processes per GPU when the number of -particles per MPI process is small (100's of particles); this can be -necessary to keep the GPU cores busy. +Using multiple MPI tasks per GPU will often give the best performance, +as allowed my most multi-core CPU/GPU configurations. -See the lammps/lib/gpu/README file for instructions on how to build -the GPU library for single, mixed, or double precision. The latter -requires that your GPU card support double precision. +If the number of particles per MPI task is small (e.g. 100s of +particles), it can be more eefficient to run with fewer MPI tasks per +GPU, even if you do not use all the cores on the compute node. + +The "Benchmark page"_http://lammps.sandia.gov/bench.html of the LAMMPS +web site gives GPU performance on a desktop machine and the Titan HPC +platform at ORNL for several of the LAMMPS benchmarks, as a function +of problem size and number of compute nodes. :line @@ -633,10 +674,303 @@ a fix or compute that is non-GPU-ized, or until output is performed (thermo or dump snapshot or restart file). The less often this occurs, the faster your simulation will run. +:line + +5.8 KOKKOS package :h4,link(acc_8) + +The KOKKOS package contains versions of pair, fix, and atom styles +that use data structures and methods and macros provided by the Kokkos +library, which is included with LAMMPS in lib/kokkos. + +"Kokkos"_http://trilinos.sandia.gov/packages/kokkos is a C++ library +that provides two key abstractions for an application like LAMMPS. +First, it allows a single implementation of an application kernel +(e.g. a pair style) to run efficiently on different kinds of hardware +(GPU, Intel Phi, many-core chip). + +Second, it provides data abstractions to adjust (at compile time) the +memory layout of basic data structures like 2d and 3d arrays and allow +the transparent utilization of special hardware load and store units. +Such data structures are used in LAMMPS to store atom coordinates or +forces or neighbor lists. The layout is chosen to optimize +performance on different platforms. Again this operation is hidden +from the developer, and does not affect how the single implementation +of the kernel is coded. + +These abstractions are set at build time, when LAMMPS is compiled with +the KOKKOS package installed. This is done by selecting a "host" and +"device" to build for, compatible with the compute nodes in your +machine. Note that if you are running on a desktop machine, you +typically have one compute node. On a cluster or supercomputer there +may be dozens or 1000s of compute nodes. The procedure for building +and running with the Kokkos library is the same, no matter how many +nodes you run on. + +All Kokkos operations occur within the context of an individual MPI +task running on a single node of the machine. The total number of MPI +tasks used by LAMMPS (one or multiple per compute node) is set in the +usual manner via the mpirun or mpiexec commands, and is independent of +Kokkos. + +Kokkos provides support for one or two modes of execution per MPI +task. This means that some computational tasks (pairwise +interactions, neighbor list builds, time integration, etc) are +parallelized in one or the other of the two modes. The first mode is +called the "host" and is one or more threads running on one or more +physical CPUs (within the node). Currently, both multi-core CPUs and +an Intel Phi processor (running in native mode) are supported. The +second mode is called the "device" and is an accelerator chip of some +kind. Currently only an NVIDIA GPU is supported. If your compute +node does not have a GPU, then there is only one mode of execution, +i.e. the host and device are the same. + +IMPORTNANT NOTE: Currently, if using GPUs, you should set the number +of MPI tasks per compute node to be equal to the number of GPUs per +compute node. In the future Kokkos will support assigning one GPU to +multiple MPI tasks or using multiple GPUs per MPI task. Currently +Kokkos does not support AMD GPUs due to limits in the available +backend programming models (in particular relative extensive C++ +support is required for the Kernel language). This is expected to +change in the future. + +Here are several examples of how to build LAMMPS and run a simulation +using the KOKKOS package for typical compute node configurations. +Note that the -np setting for the mpirun command in these examples are +for a run on a single node. To scale these examples up to run on a +system with N compute nodes, simply multiply the -np setting by N. + +All the build steps are performed from within the src directory. All +the run steps are performed in the bench directory using the in.lj +input script. It is assumed the LAMMPS executable has been copied to +that directory or whatever directory the runs are being performed in. +Details of the various options are discussed below. + +[Compute node(s) = dual hex-core CPUs and no GPU:] + +make yes-kokkos # install the KOKKOS package +make g++ OMP=yes # build with OpenMP, no CUDA :pre + +mpirun -np 12 lmp_g++ -k off < in.lj # MPI-only mode with no Kokkos +mpirun -np 12 lmp_g++ -sf kk < in.lj # MPI-only mode with Kokkos +mpirun -np 1 lmp_g++ -k on t 12 -sf kk < in.lj # one MPI task, 12 threads +mpirun -np 2 lmp_g++ -k on t 6 -sf kk < in.lj # two MPI tasks, 6 threads/task :pre + +[Compute node(s) = Intel Phi with 61 cores:] + +make yes-kokkos +make g++ OMP=yes MIC=yes # build with OpenMP for Phi :pre + +mpirun -np 12 lmp_g++ -k on t 20 -sf kk < in.lj # 12*20 = 240 total cores +mpirun -np 15 lmp_g++ -k on t 16 -sf kk < in.lj +mpirun -np 30 lmp_g++ -k on t 8 -sf kk < in.lj +mpirun -np 1 lmp_g++ -k on t 240 -sf kk < in.lj :pre + +[Compute node(s) = dual hex-core CPUs and a single GPU:] + +make yes-kokkos +make cuda CUDA=yes # build for GPU, use src/MAKE/Makefile.cuda :pre + +mpirun -np 1 lmp_cuda -k on t 6 -sf kk < in.lj :pre + +[Compute node(s) = dual 8-core CPUs and 2 GPUs:] + +make yes-kokkos +make cuda CUDA=yes :pre + +mpirun -np 2 lmp_cuda -k on t 8 g 2 -sf kk < in.lj # use both GPUs, one per MPI task :pre + +[Building LAMMPS with the KOKKOS package:] + +A summary of the build process is given here. More details and all +the available make variable options are given in "this +section"_Section_start.html#start_3_4 of the manual. + +From the src directory, type + +make yes-kokkos :pre + +to include the KOKKOS package. Then perform a normal LAMMPS build, +with additional make variable specifications to choose the host and +device you will run the resulting executable on, e.g. + +make g++ OMP=yes +make cuda CUDA=yes :pre + +As illustrated above, the most important variables to set are OMP, +CUDA, and MIC. The default settings are OMP=yes, CUDA=no, MIC=no +Setting OMP to {yes} will use OpenMP for threading on the host, as +well as on the device (if no GPU is present). Setting CUDA to {yes} +will use one or more GPUs as the device. Setting MIC=yes is necessary +when building for an Intel Phi processor. + +Note that to use a GPU, you must use a lo-level Makefile, +e.g. src/MAKE/Makefile.cuda as included in the LAMMPS distro, which +uses the NVIDA "nvcc" compiler. You must check that the CCFLAGS -arch +setting is appropriate for your NVIDIA hardware and installed +software. Typical values for -arch are given in "this +section"_Section_start.html#start_3_4 of the manual, as well as other +settings that must be included in the lo-level Makefile, if you create +your own. + +[Input scripts and use of command-line switches -kokkos and -suffix:] + +To use any Kokkos-enabled style provided in the KOKKOS package, you +must use a Kokkos-enabled atom style. LAMMPS will give an error if +you do not do this. + +There are two command-line switches relevant to using Kokkos, -k or +-kokkos, and -sf or -suffix. They are described in detail in "this +section"_Section_start.html#start_7 of the manual. + +Here are common options to use: + +-k off : runs an executable built with the KOKKOS pacakage, as + if Kokkos were not installed. :ulb,l + +-sf kk : enables automatic use of Kokkos versions of atom, pair, +fix, compute styles if they exist. This can also be done with more +precise control by using the "suffix"_suffix.html command or appending +"kk" to styles within the input script, e.g. "pair_style lj/cut/kk". :l + +-k on t Nt : specifies how many threads per MPI task to use within a + compute node. For good performance, the product of MPI tasks * + threads/task should not exceed the number of physical CPU or Intel + Phi cores. :l + +-k on g Ng : specifies how many GPUs per compute node are available. +The default is 1, so this should be specified is you have 2 or more +GPUs per compute node. :ule,l + +[Use of package command options:] + +Using the "package kokkos"_package.html command in an input script +allows choice of options for neighbor lists and communication. See +the "package"_package.html command doc page for details and default +settings. + +Experimenting with different styles of neighbor lists or inter-node +communication can provide a speed-up for specific calculations. + +[Running on a multi-core CPU:] + +Build with OMP=yes (the default) and CUDA=no (the default). + +If N is the number of physical cores/node, then the number of MPI +tasks/node * number of threads/task should not exceed N, and should +typically equal N. Note that the default threads/task is 1, as set by +the "t" keyword of the -k "command-line +switch"_Section_start.html#start_7. If you do not change this, no +additional parallelism (beyond MPI) will be invoked on the host +CPU(s). + +You can compare the performance running in different modes: + +run with 1 MPI task/node and N threads/task +run with N MPI tasks/node and 1 thread/task +run with settings in between these extremes :ul + +Examples of mpirun commands in these modes, for nodes with dual +hex-core CPUs and no GPU, are shown above. + +[Running on GPUs:] + +Build with CUDA=yes, using src/MAKE/Makefile.cuda. Insure the setting +for CUDA_PATH in lib/kokkos/Makefile.lammps is correct for your Cuda +software installation. Insure the -arch setting in +src/MAKE/Makefile.cuda is correct for your GPU hardware/software (see +"this section"_Section_start.html#start_3_4 of the manual for details. + +The -np setting of the mpirun command should set the number of MPI +tasks/node to be equal to the # of physical GPUs on the node. + +Use the "-kokkos command-line switch"_Section_commands.html#start_7 to +specify the number of GPUs per node, and the number of threads per MPI +task. As above for multi-core CPUs (and no GPU), if N is the number +of physical cores/node, then the number of MPI tasks/node * number of +threads/task should not exceed N. With one GPU (and one MPI task) it +may be faster to use less than all the available cores, by setting +threads/task to a smaller value. This is because using all the cores +on a dual-socket node will incur extra cost to copy memory from the +2nd socket to the GPU. + +Examples of mpirun commands that follow these rules, for nodes with +dual hex-core CPUs and one or two GPUs, are shown above. + +[Running on an Intel Phi:] + +Kokkos only uses Intel Phi processors in their "native" mode, i.e. +not hosted by a CPU. + +Build with OMP=yes (the default) and MIC=yes. The latter +insures code is correctly compiled for the Intel Phi. The +OMP setting means OpenMP will be used for parallelization +on the Phi, which is currently the best option within +Kokkos. In the future, other options may be added. + +Current-generation Intel Phi chips have either 61 or 57 cores. One +core should be excluded to run the OS, leaving 60 or 56 cores. Each +core is hyperthreaded, so there are effectively N = 240 (4*60) or N = +224 (4*56) cores to run on. + +The -np setting of the mpirun command sets the number of MPI +tasks/node. The "-k on t Nt" command-line switch sets the number of +threads/task as Nt. The product of these 2 values should be N, i.e. +240 or 224. Also, the number of threads/task should be a multiple of +4 so that logical threads from more than one MPI task do not run on +the same physical core. + +Examples of mpirun commands that follow these rules, for Intel Phi +nodes with 61 cores, are shown above. + +[Examples and benchmarks:] + +The examples/kokkos and bench/KOKKOS directories have scripts that can +be run with the KOKKOS package, as well as detailed instructions on +how to run them. + +IMPORTANT NOTE: the bench/KOKKOS directory does not yet exist. It +will be added later. + +[Additional performance issues:] + +When using threads (OpenMP or pthreads), it is important for +performance to bind the threads to physical cores, so they do not +migrate during a simulation. The same is true for MPI tasks, but the +default binding rules implemented for various MPI versions, do not +account for thread binding. + +Thus if you use more than one thread per MPI task, you should insure +MPI tasks are bound to CPU sockets. Furthermore, use thread affinity +environment variables from the OpenMP runtime when using OpenMP and +compile with hwloc support when using pthreads. With OpenMP 3.1 (gcc +4.7 or later, intel 12 or later) setting the environment variable +OMP_PROC_BIND=true should be sufficient. A typical mpirun command +should set these flags: + +OpenMPI 1.8: mpirun -np 2 -bind-to socket -map-by socket ./lmp_openmpi ... +Mvapich2 2.0: mpiexec -np 2 -bind-to socket -map-by socket ./lmp_mvapich ... :pre + +When using a GPU, you will achieve the best performance if your input +script does not use any fix or compute styles which are not yet +Kokkos-enabled. This allows data to stay on the GPU for multiple +timesteps, without being copied back to the host CPU. Invoking a +non-Kokkos fix or compute, or performing I/O for +"thermo"_thermo_style.html or "dump"_dump.html output will cause data +to be copied back to the CPU. + +You cannot yet assign multiple MPI tasks to the same GPU with the +KOKKOS package. We plan to support this in the future, similar to the +GPU package in LAMMPS. + +You cannot yet use both the host (multi-threaded) and device (GPU) +together to compute pairwise interactions with the KOKKOS package. We +hope to support this in the future, similar to the GPU package in +LAMMPS. + :line :line -5.8 Comparison of GPU and USER-CUDA packages :h4,link(acc_8) +5.9 Comparison of GPU and USER-CUDA packages :h4,link(acc_9) Both the GPU and USER-CUDA packages accelerate a LAMMPS calculation using NVIDIA hardware, but they do it in different ways. diff --git a/doc/Section_commands.html b/doc/Section_commands.html index 085743c918..c8f264577b 100644 --- a/doc/Section_commands.html +++ b/doc/Section_commands.html @@ -428,10 +428,10 @@ package.
(3) The KIM package was created by Valeriu Smirichinski, Ryan Elliott, and Ellad Tadmor (U Minn).
+(4) The KOKKOS package was created primarily by Christian Trott +(Sandia). It uses the Kokkos library which was developed by Carter +Edwards, Christian, and collaborators at Sandia. +
The "Doc page" column links to either a portion of the Section_howto of the manual, or an input script command implemented as part of the package. diff --git a/doc/Section_packages.txt b/doc/Section_packages.txt index 700c88b951..7ede067fd8 100644 --- a/doc/Section_packages.txt +++ b/doc/Section_packages.txt @@ -45,15 +45,16 @@ CLASS2, class 2 force fields, -, "pair_style lj/class2"_pair_class2.html, -, - COLLOID, colloidal particles, -, "atom_style colloid"_atom_style.html, colloid, - DIPOLE, point dipole particles, -, "pair_style dipole/cut"_pair_dipole.html, dipole, - FLD, Fast Lubrication Dynamics, Kumar & Bybee & Higdon (1), "pair_style lubricateU"_pair_lubricateU.html, -, - -GPU, GPU-enabled potentials, Mike Brown (ORNL), "Section accelerate"_Section_accelerate.html#acc_6, gpu, lib/gpu +GPU, GPU-enabled styles, Mike Brown (ORNL), "Section accelerate"_Section_accelerate.html#acc_6, gpu, lib/gpu GRANULAR, granular systems, -, "Section_howto"_Section_howto.html#howto_6, pour, - KIM, openKIM potentials, Smirichinski & Elliot & Tadmor (3), "pair_style kim"_pair_kim.html, kim, KIM +KOKKOS, Kokkos-enabled styles, Trott & Edwards (4), "Section_accelerate"_Section_accelerate.html#acc_8, kokkos, lib/kokkos KSPACE, long-range Coulombic solvers, -, "kspace_style"_kspace_style.html, peptide, - MANYBODY, many-body potentials, -, "pair_style tersoff"_pair_tersoff.html, shear, - MEAM, modified EAM potential, Greg Wagner (Sandia), "pair_style meam"_pair_meam.html, meam, lib/meam MC, Monte Carlo options, -, "fix gcmc"_fix_gcmc.html, -, - MOLECULE, molecular system force fields, -, "Section_howto"_Section_howto.html#howto_3, peptide, - -OPT, optimized pair potentials, Fischer & Richie & Natoli (2), "Section accelerate"_Section_accelerate.html#acc_4, -, - +OPT, optimized pair styles, Fischer & Richie & Natoli (2), "Section accelerate"_Section_accelerate.html#acc_4, -, - PERI, Peridynamics models, Mike Parks (Sandia), "pair_style peri"_pair_peri.html, peri, - POEMS, coupled rigid body motion, Rudra Mukherjee (JPL), "fix poems"_fix_poems.html, rigid, lib/poems REAX, ReaxFF potential, Aidan Thompson (Sandia), "pair_style reax"_pair_reax.html, reax, lib/reax @@ -78,6 +79,10 @@ Technolgy). (3) The KIM package was created by Valeriu Smirichinski, Ryan Elliott, and Ellad Tadmor (U Minn). +(4) The KOKKOS package was created primarily by Christian Trott +(Sandia). It uses the Kokkos library which was developed by Carter +Edwards, Christian, and collaborators at Sandia. + The "Doc page" column links to either a portion of the "Section_howto"_Section_howto.html of the manual, or an input script command implemented as part of the package. diff --git a/doc/Section_start.html b/doc/Section_start.html index f8c34710b0..12abe89a3f 100644 --- a/doc/Section_start.html +++ b/doc/Section_start.html @@ -555,7 +555,7 @@ on both a basic build and a customized build with pacakges you select.
Code for some of these auxiliary libraries is included in the LAMMPS distribution under the lib directory. Examples are the USER-ATC and -MEAM packages. Some auxiliary libraries are not included with LAMMPS; +MEAM packages. Some auxiliary libraries are NOT included with LAMMPS; to use the associated package you must download and install the auxiliary library yourself. Examples are the KIM and VORONOI and USER-MOLFILE packages. @@ -699,31 +699,33 @@ that library. Typically this is done by typing something like:
make -f Makefile.g++-
If one of the provided Makefiles is not -appropriate for your system you will need to edit or add one. -Note that all the Makefiles have a setting for EXTRAMAKE at -the top that names a Makefile.lammps.* file. +
If one of the provided Makefiles is not appropriate for your system +you will need to edit or add one. Note that all the Makefiles have a +setting for EXTRAMAKE at the top that specifies a Makefile.lammps.* +file.
-If successful, this will produce 2 files in the lib directory: +
If the library build is successful, it will produce 2 files in the lib +directory:
libpackage.a Makefile.lammps-
The Makefile.lammps file is a copy of the EXTRAMAKE file specified -in the Makefile you used. +
The Makefile.lammps file will be a copy of the EXTRAMAKE file setting +specified in the library Makefile.* you used.
-You MUST insure that the settings in Makefile.lammps are appropriate -for your system. If they are not, the LAMMPS build will fail. +
Note that you must insure that the settings in Makefile.lammps are +appropriate for your system. If they are not, the LAMMPS build will +fail.
-As explained in the lib/package/README files, they are used to specify -additional system libraries and their locations so that LAMMPS can -build with the auxiliary library. For example, if the MEAM or REAX -packages are used, the auxiliary libraries consist of F90 code, build -with a F90 complier. To link that library with LAMMPS (a C++ code) -via whatever C++ compiler LAMMPS is built with, typically requires -additional Fortran-to-C libraries be included in the link. Another -example are the BLAS and LAPACK libraries needed to use the USER-ATC -or USER-AWPMD packages. +
As explained in the lib/package/README files, the settings in +Makefile.lammps are used to specify additional system libraries and +their locations so that LAMMPS can build with the auxiliary library. +For example, if the MEAM or REAX packages are used, the auxiliary +libraries consist of F90 code, built with a Fortran complier. To link +that library with LAMMPS (a C++ code) via whatever C++ compiler LAMMPS +is built with, typically requires additional Fortran-to-C libraries be +included in the link. Another example are the BLAS and LAPACK +libraries needed to use the USER-ATC or USER-AWPMD packages.
For libraries without provided source code, see the src/package/Makefile.lammps file for information on where to find the @@ -731,10 +733,105 @@ library and how to build it. E.g. the file src/KIM/Makefile.lammps or src/VORONOI/Makefile.lammps or src/UESR-MOLFILE/Makefile.lammps. These files serve the same purpose as the lib/package/Makefile.lammps files described above. The files have settings needed when LAMMPS is -built to link with the corresponding auxiliary library. Again, you -MUST insure that the settings in src/package/Makefile.lammps are -appropriate for your system and where you installed the auxiliary -library. If they are not, the LAMMPS build will fail. +built to link with the corresponding auxiliary library. +
+Again, you must insure that the settings in +src/package/Makefile.lammps are appropriate for your system and where +you installed the auxiliary library. If they are not, the LAMMPS +build will fail. +
+One package, the KOKKOS package, allows its build options to be +specified by setting variables via the "make" command, rather than by +first building an auxiliary library and editing a Makefile.lammps +file, as discussed in the previous sub-section for other packages. +This is for convenience since it is common to want to experiment with +different Kokkos library options. Using variables enables a direct +re-build of LAMMPS and its Kokkos dependencies, so that a benchmark +test with different Kokkos options can be quickly performed. +
+The syntax for setting make variables is as follows. You must +use a GNU-compatible make command for this to work. Try "gmake" +if your system's standard make complains. +
+make yes-kokkos +make g++ VAR1=value VAR2=value ... ++
The first line installs the KOKKOS package, which only needs to be +done once. The second line builds LAMMPS with src/MAKE/Makefile.g++ +and optionally sets one or more variables that affect the build. Each +variable is specified in upper-case; its value follows an equal sign +with no spaces. The second line can be repeated with different +variable settings, though a "clean" must be done before the rebuild. +Type "make clean" to see options for this operation. +
+These are the variables that can be specified. Each takes a value of +yes or no. The default value is listed, which is set in the +lib/kokkos/Makefile.lammps file. See this +section for a discussion of what is +meant by "host" and "device" in the Kokkos context. +
+OMP sets the parallelization method used for Kokkos code (within +LAMMPS) that runs on the host. OMP=yes means that OpenMP will be +used. OMP=no means that pthreads will be used. +
+CUDA sets the parallelization method used for Kokkos code (within +LAMMPS) that runs on the device. CUDA=yes means an NVIDIA GPU running +CUDA will be used. CUDA=no means that the OMP=yes or OMP=no setting +will be used for the device as well as the host. +
+If CUDA=yes, then the lo-level Makefile in the src/MAKE directory must +use "nvcc" as its compiler, via its CC setting. For best performance +its CCFLAGS setting should use -O3 and have an -arch setting that +matches the compute capability of your NVIDIA hardware and software +installation, e.g. -arch=sm_20. Generally Fermi Generation GPUs are +sm_20, while Kepler generation GPUs are sm_30 or sm_35 and Maxwell +cards are sm_50. A complete list can be found on +wikipedia. You can +also use the deviceQuery tool that comes with the CUDA samples. Note +the minimal required compute capability is 2.0, but this will give +signicantly reduced performance compared to Kepler generation GPUs +with compute capability 3.x. For the LINK setting, "nvcc" should not +be used; instead use g++ or another compiler suitable for linking C++ +applications. Often you will want to use your MPI compiler wrapper +for this setting (i.e. mpicxx). Finally, the lo-level Makefile must +also have a "Compilation rule" for creating *.o files from *.cu files. +See src/Makefile.cuda for an example of a lo-level Makefile with all +of these settings. +
+HWLOC binds threads to hardware cores, so they do not migrate during a +simulation. HWLOC=yes should always be used if running with OMP=no +for pthreads. It is not necessary for OMP=yes for OpenMP, because +OpenMP provides alternative methods via environment variables for +binding threads to hardware cores. More info on binding threads to +cores is given in this section. +
+AVX enables Intel advanced vector extensions when compiling for an +Intel-compatible chip. AVX=yes should only be set if your host +hardware supports AVX. If it does not support it, this will cause a +run-time crash. +
+MIC enables compiler switches needed when compling for an Intel Phi +processor. +
+LIBRT enables use of a more accurate timer mechanism on most Unix +platforms. This library is not available on all platforms. +
+DEBUG is only useful when developing a Kokkos-enabled style within +LAMMPS. DEBUG=yes enables printing of run-time debugging information +that can be useful. It also enables runtime bounds checking on Kokkos +data structures.
-kokkos on/off keyword/value ... ++
Explicitly enable or disable Kokkos support, as provided by the KOKKOS +package. If LAMMPS is built with this package, as described above in +Section 2.3, then by default LAMMPS will run in Kokkos +mode. If this switch is set to "off", then it will not, even if it +was built with the KOKKOS package, which means you can run standard +LAMMPS styles or use styles enhanced by other acceleration packages, +such as the GPU or USER-CUDA or USER-OMP packages, for testing or +benchmarking purposes. The only reason to set the switch to "on", is +to check if LAMMPS was built with the KOKKOS package, since an error +will be generated if it was not. +
+Additional optional keyword/value pairs can be specified which +determine how Kokkos will use the underlying hardware on your +platform. These settings apply to each MPI task you launch via the +"mpirun" or "mpiexec" command. You may choose to run one or more MPI +tasks per physical node. Note that if you are running on a desktop +machine, you typically have one physical node. On a cluster or +supercomputer there may be dozens or 1000s of physical nodes. +
+Either the full word or an abbreviation can be used for the keywords. +Note that the keywords do not use a leading minus sign. I.e. the +keyword is "t", not "-t". Also note that each of the keywords has a +default setting. More explanation as to when to use these options and +what settings to use on different platforms is given in this +section. +
+device Nd ++
This option is only relevant if you built LAMMPS with CUDA=yes, you +have more than one GPU per node, and if you are running with only one +MPI task per node. The Nd setting is the ID of the GPU on the node to +run on. By default Nd = 0. If you have multiple GPUs per node, they +have consecutive IDs numbered as 0,1,2,etc. This setting allows you +to launch multiple independent jobs on the node, each with a single +MPI task per node, and assign each job to run on a different GPU. +
+gpus Ng Ns ++
This option is only relevant if you built LAMMPS with CUDA=yes, you +have more than one GPU per node, and you are running with multiple MPI +tasks per node (up to one per GPU). The Ng setting is how many GPUs +you will use. The Ns setting is optional. If set, it is the ID of a +GPU to skip when assigning MPI tasks to GPUs. This may be useful if +your desktop system reserves one GPU to drive the screen and the rest +are intended for computational work like running LAMMPS. By default +Ng = 1 and Ns is not set. +
+Depending on which flavor of MPI you are running, LAMMPS will look for +one of these 3 environment variables +
+SLURM_LOCALID (various MPI variants compiled with SLURM support) +MV2_COMM_WORLD_LOCAL_RANK (Mvapich) +OMPI_COMM_WORLD_LOCAL_RANK (OpenMPI) ++
which are initialized by the "srun", "mpirun" or "mpiexec" commands. +The environment variable setting for each MPI rank is used to assign a +unique GPU ID to the MPI task. +
+threads Nt ++
This option assigns Nt number of threads to each MPI task for +performing work when Kokkos is executing in OpenMP or pthreads mode. +The default is Nt = 1, which essentially runs in MPI-only mode. If +there are Np MPI tasks per physical node, you generally want Np*Nt = +the number of physical cores per node, to use your available hardware +optimally. This also sets the number of threads used by the host when +LAMMPS is compiled with CUDA=yes. +
+numa Nm ++
This option is only relevant when using pthreads with hwloc support. +In this case Nm defines the number of NUMA regions (typicaly sockets) +on a node which will be utilizied by a single MPI rank. By default Nm += 1. If this option is used the total number of worker-threads per +MPI rank is threads*numa. Currently it is always almost better to +assign at least one MPI rank per NUMA region, and leave numa set to +its default value of 1. This is because letting a single process span +multiple NUMA regions induces a significant amount of cross NUMA data +traffic which is slow. +
-log file
Specify a log file for LAMMPS to write status information to. In @@ -1277,23 +1462,24 @@ multi-partition mode, if the specified file is "none", then no screen output is performed. Option -pscreen will override the name of the partition screen files file.N.
--suffix style ++-suffix style argsUse variants of various styles if they exist. The specified style can -be opt, omp, gpu, or cuda. These refer to optional packages that -LAMMPS can be built with, as described above in Section -2.3. The "opt" style corrsponds to the OPT package, the -"omp" style to the USER-OMP package, the "gpu" style to the GPU -package, and the "cuda" style to the USER-CUDA package. +be cuda, gpu, kk, omp, or opt. These refer to optional +packages that LAMMPS can be built with, as described above in Section +2.3. The "cuda" style corresponds to the USER-CUDA package, +the "gpu" style to the GPU package, the "kk" style to the KOKKOS +pacakge, the "opt" style to the OPT package, and the "omp" style to +the USER-OMP package.
As an example, all of the packages provide a pair_style -lj/cut variant, with style names lj/cut/opt, lj/cut/omp, -lj/cut/gpu, or lj/cut/cuda. A variant styles can be specified -explicitly in your input script, e.g. pair_style lj/cut/gpu. If the --suffix switch is used, you do not need to modify your input script. -The specified suffix (opt,omp,gpu,cuda) is automatically appended -whenever your input script command creates a new -atom, pair, fix, +lj/cut variant, with style names lj/cut/cuda, +lj/cut/gpu, lj/cut/kk, lj/cut/omp, or lj/cut/opt. A variant styles +can be specified explicitly in your input script, e.g. pair_style +lj/cut/gpu. If the -suffix switch is used, you do not need to modify +your input script. The specified suffix (cuda,gpu,kk,omp,opt) is +automatically appended whenever your input script command creates a +new atom, pair, fix, compute, or run style. If the variant version does not exist, the standard version is created.
@@ -1303,13 +1489,20 @@ default GPU settings, as if the command "package gpu force/neigh 0 0 changed by using the package gpu command in your script if desired. +For the KOKKOS package, using this command-line switch also invokes +the default KOKKOS settings, as if the command "package kokkos neigh +full comm/exchange host comm/forward host " were used at the top of +your input script. These settings can be changed by using the +package kokkos command in your script if desired. +
For the OMP package, using this command-line switch also invokes the default OMP settings, as if the command "package omp *" were used at the top of your input script. These settings can be changed by using the package omp command in your script if desired.
-The suffix command can also set a suffix and it can also -turn off/on any suffix setting made via the command line. +
The suffix command can also be used set a suffix and it +can also turn off or back on any suffix setting made via the command +line.
-var name value1 value2 ...diff --git a/doc/Section_start.txt b/doc/Section_start.txt index c126503b1f..2e8d786809 100644 --- a/doc/Section_start.txt +++ b/doc/Section_start.txt @@ -549,7 +549,7 @@ This section has the following sub-sections: "Package basics"_#start_3_1 "Including/excluding packages"_#start_3_2 "Packages that require extra libraries"_#start_3_3 -"Additional Makefile settings for extra libraries"_#start_3_4 :ul +"Packages that use make variable settings"_#start_3_4 :ul :line @@ -682,7 +682,7 @@ for a list of packages that have auxiliary libraries. Code for some of these auxiliary libraries is included in the LAMMPS distribution under the lib directory. Examples are the USER-ATC and -MEAM packages. Some auxiliary libraries are not included with LAMMPS; +MEAM packages. Some auxiliary libraries are NOT included with LAMMPS; to use the associated package you must download and install the auxiliary library yourself. Examples are the KIM and VORONOI and USER-MOLFILE packages. @@ -693,31 +693,33 @@ that library. Typically this is done by typing something like: make -f Makefile.g++ :pre -If one of the provided Makefiles is not -appropriate for your system you will need to edit or add one. -Note that all the Makefiles have a setting for EXTRAMAKE at -the top that names a Makefile.lammps.* file. +If one of the provided Makefiles is not appropriate for your system +you will need to edit or add one. Note that all the Makefiles have a +setting for EXTRAMAKE at the top that specifies a Makefile.lammps.* +file. -If successful, this will produce 2 files in the lib directory: +If the library build is successful, it will produce 2 files in the lib +directory: libpackage.a Makefile.lammps :pre -The Makefile.lammps file is a copy of the EXTRAMAKE file specified -in the Makefile you used. +The Makefile.lammps file will be a copy of the EXTRAMAKE file setting +specified in the library Makefile.* you used. -You MUST insure that the settings in Makefile.lammps are appropriate -for your system. If they are not, the LAMMPS build will fail. +Note that you must insure that the settings in Makefile.lammps are +appropriate for your system. If they are not, the LAMMPS build will +fail. -As explained in the lib/package/README files, they are used to specify -additional system libraries and their locations so that LAMMPS can -build with the auxiliary library. For example, if the MEAM or REAX -packages are used, the auxiliary libraries consist of F90 code, build -with a F90 complier. To link that library with LAMMPS (a C++ code) -via whatever C++ compiler LAMMPS is built with, typically requires -additional Fortran-to-C libraries be included in the link. Another -example are the BLAS and LAPACK libraries needed to use the USER-ATC -or USER-AWPMD packages. +As explained in the lib/package/README files, the settings in +Makefile.lammps are used to specify additional system libraries and +their locations so that LAMMPS can build with the auxiliary library. +For example, if the MEAM or REAX packages are used, the auxiliary +libraries consist of F90 code, built with a Fortran complier. To link +that library with LAMMPS (a C++ code) via whatever C++ compiler LAMMPS +is built with, typically requires additional Fortran-to-C libraries be +included in the link. Another example are the BLAS and LAPACK +libraries needed to use the USER-ATC or USER-AWPMD packages. For libraries without provided source code, see the src/package/Makefile.lammps file for information on where to find the @@ -725,10 +727,105 @@ library and how to build it. E.g. the file src/KIM/Makefile.lammps or src/VORONOI/Makefile.lammps or src/UESR-MOLFILE/Makefile.lammps. These files serve the same purpose as the lib/package/Makefile.lammps files described above. The files have settings needed when LAMMPS is -built to link with the corresponding auxiliary library. Again, you -MUST insure that the settings in src/package/Makefile.lammps are -appropriate for your system and where you installed the auxiliary -library. If they are not, the LAMMPS build will fail. +built to link with the corresponding auxiliary library. + +Again, you must insure that the settings in +src/package/Makefile.lammps are appropriate for your system and where +you installed the auxiliary library. If they are not, the LAMMPS +build will fail. + +:line + +[{Packages that use make variable settings}] :link(start_3_4) + +One package, the KOKKOS package, allows its build options to be +specified by setting variables via the "make" command, rather than by +first building an auxiliary library and editing a Makefile.lammps +file, as discussed in the previous sub-section for other packages. +This is for convenience since it is common to want to experiment with +different Kokkos library options. Using variables enables a direct +re-build of LAMMPS and its Kokkos dependencies, so that a benchmark +test with different Kokkos options can be quickly performed. + +The syntax for setting make variables is as follows. You must +use a GNU-compatible make command for this to work. Try "gmake" +if your system's standard make complains. + +make yes-kokkos +make g++ VAR1=value VAR2=value ... :pre + +The first line installs the KOKKOS package, which only needs to be +done once. The second line builds LAMMPS with src/MAKE/Makefile.g++ +and optionally sets one or more variables that affect the build. Each +variable is specified in upper-case; its value follows an equal sign +with no spaces. The second line can be repeated with different +variable settings, though a "clean" must be done before the rebuild. +Type "make clean" to see options for this operation. + +These are the variables that can be specified. Each takes a value of +{yes} or {no}. The default value is listed, which is set in the +lib/kokkos/Makefile.lammps file. See "this +section"_Section_accelerate.html#acc_8 for a discussion of what is +meant by "host" and "device" in the Kokkos context. + +OMP, default = {yes} +CUDA, default = {no} +HWLOC, default = {no} +AVX, default = {no} +MIC, default = {no} +LIBRT, default = {no} +DEBUG, default = {no} :ul + +OMP sets the parallelization method used for Kokkos code (within +LAMMPS) that runs on the host. OMP=yes means that OpenMP will be +used. OMP=no means that pthreads will be used. + +CUDA sets the parallelization method used for Kokkos code (within +LAMMPS) that runs on the device. CUDA=yes means an NVIDIA GPU running +CUDA will be used. CUDA=no means that the OMP=yes or OMP=no setting +will be used for the device as well as the host. + +If CUDA=yes, then the lo-level Makefile in the src/MAKE directory must +use "nvcc" as its compiler, via its CC setting. For best performance +its CCFLAGS setting should use -O3 and have an -arch setting that +matches the compute capability of your NVIDIA hardware and software +installation, e.g. -arch=sm_20. Generally Fermi Generation GPUs are +sm_20, while Kepler generation GPUs are sm_30 or sm_35 and Maxwell +cards are sm_50. A complete list can be found on +"wikipedia"_http://en.wikipedia.org/wiki/CUDA#Supported_GPUs. You can +also use the deviceQuery tool that comes with the CUDA samples. Note +the minimal required compute capability is 2.0, but this will give +signicantly reduced performance compared to Kepler generation GPUs +with compute capability 3.x. For the LINK setting, "nvcc" should not +be used; instead use g++ or another compiler suitable for linking C++ +applications. Often you will want to use your MPI compiler wrapper +for this setting (i.e. mpicxx). Finally, the lo-level Makefile must +also have a "Compilation rule" for creating *.o files from *.cu files. +See src/Makefile.cuda for an example of a lo-level Makefile with all +of these settings. + +HWLOC binds threads to hardware cores, so they do not migrate during a +simulation. HWLOC=yes should always be used if running with OMP=no +for pthreads. It is not necessary for OMP=yes for OpenMP, because +OpenMP provides alternative methods via environment variables for +binding threads to hardware cores. More info on binding threads to +cores is given in "this section"_Section_accelerate.html#acc_8. + +AVX enables Intel advanced vector extensions when compiling for an +Intel-compatible chip. AVX=yes should only be set if your host +hardware supports AVX. If it does not support it, this will cause a +run-time crash. + +MIC enables compiler switches needed when compling for an Intel Phi +processor. + +LIBRT enables use of a more accurate timer mechanism on most Unix +platforms. This library is not available on all platforms. + +DEBUG is only useful when developing a Kokkos-enabled style within +LAMMPS. DEBUG=yes enables printing of run-time debugging information +that can be useful. It also enables runtime bounds checking on Kokkos +data structures. :line @@ -1038,6 +1135,7 @@ letter abbreviation can be used: -e or -echo -i or -in -h or -help +-k or -kokkos -l or -log -nc or -nocite -p or -partition @@ -1098,6 +1196,93 @@ want to use was included via the appropriate package at compile time. LAMMPS will print the info and immediately exit if this switch is used. +-kokkos on/off keyword/value ... :pre + +Explicitly enable or disable Kokkos support, as provided by the KOKKOS +package. If LAMMPS is built with this package, as described above in +"Section 2.3"_#start_3, then by default LAMMPS will run in Kokkos +mode. If this switch is set to "off", then it will not, even if it +was built with the KOKKOS package, which means you can run standard +LAMMPS styles or use styles enhanced by other acceleration packages, +such as the GPU or USER-CUDA or USER-OMP packages, for testing or +benchmarking purposes. The only reason to set the switch to "on", is +to check if LAMMPS was built with the KOKKOS package, since an error +will be generated if it was not. + +Additional optional keyword/value pairs can be specified which +determine how Kokkos will use the underlying hardware on your +platform. These settings apply to each MPI task you launch via the +"mpirun" or "mpiexec" command. You may choose to run one or more MPI +tasks per physical node. Note that if you are running on a desktop +machine, you typically have one physical node. On a cluster or +supercomputer there may be dozens or 1000s of physical nodes. + +Either the full word or an abbreviation can be used for the keywords. +Note that the keywords do not use a leading minus sign. I.e. the +keyword is "t", not "-t". Also note that each of the keywords has a +default setting. More explanation as to when to use these options and +what settings to use on different platforms is given in "this +section"_Section_accerlerate.html#acc_8. + +d or device +g or gpus +t or threads +n or numa :ul + +device Nd :pre + +This option is only relevant if you built LAMMPS with CUDA=yes, you +have more than one GPU per node, and if you are running with only one +MPI task per node. The Nd setting is the ID of the GPU on the node to +run on. By default Nd = 0. If you have multiple GPUs per node, they +have consecutive IDs numbered as 0,1,2,etc. This setting allows you +to launch multiple independent jobs on the node, each with a single +MPI task per node, and assign each job to run on a different GPU. + +gpus Ng Ns :pre + +This option is only relevant if you built LAMMPS with CUDA=yes, you +have more than one GPU per node, and you are running with multiple MPI +tasks per node (up to one per GPU). The Ng setting is how many GPUs +you will use. The Ns setting is optional. If set, it is the ID of a +GPU to skip when assigning MPI tasks to GPUs. This may be useful if +your desktop system reserves one GPU to drive the screen and the rest +are intended for computational work like running LAMMPS. By default +Ng = 1 and Ns is not set. + +Depending on which flavor of MPI you are running, LAMMPS will look for +one of these 3 environment variables + +SLURM_LOCALID (various MPI variants compiled with SLURM support) +MV2_COMM_WORLD_LOCAL_RANK (Mvapich) +OMPI_COMM_WORLD_LOCAL_RANK (OpenMPI) :pre + +which are initialized by the "srun", "mpirun" or "mpiexec" commands. +The environment variable setting for each MPI rank is used to assign a +unique GPU ID to the MPI task. + +threads Nt :pre + +This option assigns Nt number of threads to each MPI task for +performing work when Kokkos is executing in OpenMP or pthreads mode. +The default is Nt = 1, which essentially runs in MPI-only mode. If +there are Np MPI tasks per physical node, you generally want Np*Nt = +the number of physical cores per node, to use your available hardware +optimally. This also sets the number of threads used by the host when +LAMMPS is compiled with CUDA=yes. + +numa Nm :pre + +This option is only relevant when using pthreads with hwloc support. +In this case Nm defines the number of NUMA regions (typicaly sockets) +on a node which will be utilizied by a single MPI rank. By default Nm += 1. If this option is used the total number of worker-threads per +MPI rank is threads*numa. Currently it is always almost better to +assign at least one MPI rank per NUMA region, and leave numa set to +its default value of 1. This is because letting a single process span +multiple NUMA regions induces a significant amount of cross NUMA data +traffic which is slow. + -log file :pre Specify a log file for LAMMPS to write status information to. In @@ -1271,23 +1456,24 @@ multi-partition mode, if the specified file is "none", then no screen output is performed. Option -pscreen will override the name of the partition screen files file.N. --suffix style :pre +-suffix style args :pre Use variants of various styles if they exist. The specified style can -be {opt}, {omp}, {gpu}, or {cuda}. These refer to optional packages that -LAMMPS can be built with, as described above in "Section -2.3"_#start_3. The "opt" style corrsponds to the OPT package, the -"omp" style to the USER-OMP package, the "gpu" style to the GPU -package, and the "cuda" style to the USER-CUDA package. +be {cuda}, {gpu}, {kk}, {omp}, or {opt}. These refer to optional +packages that LAMMPS can be built with, as described above in "Section +2.3"_#start_3. The "cuda" style corresponds to the USER-CUDA package, +the "gpu" style to the GPU package, the "kk" style to the KOKKOS +pacakge, the "opt" style to the OPT package, and the "omp" style to +the USER-OMP package. As an example, all of the packages provide a "pair_style -lj/cut"_pair_lj.html variant, with style names lj/cut/opt, lj/cut/omp, -lj/cut/gpu, or lj/cut/cuda. A variant styles can be specified -explicitly in your input script, e.g. pair_style lj/cut/gpu. If the --suffix switch is used, you do not need to modify your input script. -The specified suffix (opt,omp,gpu,cuda) is automatically appended -whenever your input script command creates a new -"atom"_atom_style.html, "pair"_pair_style.html, "fix"_fix.html, +lj/cut"_pair_lj.html variant, with style names lj/cut/cuda, +lj/cut/gpu, lj/cut/kk, lj/cut/omp, or lj/cut/opt. A variant styles +can be specified explicitly in your input script, e.g. pair_style +lj/cut/gpu. If the -suffix switch is used, you do not need to modify +your input script. The specified suffix (cuda,gpu,kk,omp,opt) is +automatically appended whenever your input script command creates a +new "atom"_atom_style.html, "pair"_pair_style.html, "fix"_fix.html, "compute"_compute.html, or "run"_run_style.html style. If the variant version does not exist, the standard version is created. @@ -1297,13 +1483,20 @@ default GPU settings, as if the command "package gpu force/neigh 0 0 changed by using the "package gpu"_package.html command in your script if desired. +For the KOKKOS package, using this command-line switch also invokes +the default KOKKOS settings, as if the command "package kokkos neigh +full comm/exchange host comm/forward host " were used at the top of +your input script. These settings can be changed by using the +"package kokkos"_package.html command in your script if desired. + For the OMP package, using this command-line switch also invokes the default OMP settings, as if the command "package omp *" were used at the top of your input script. These settings can be changed by using the "package omp"_package.html command in your script if desired. -The "suffix"_suffix.html command can also set a suffix and it can also -turn off/on any suffix setting made via the command line. +The "suffix"_suffix.html command can also be used set a suffix and it +can also turn off or back on any suffix setting made via the command +line. -var name value1 value2 ... :pre diff --git a/doc/atom_style.html b/doc/atom_style.html index d7af8e203f..0b606843af 100644 --- a/doc/atom_style.html +++ b/doc/atom_style.html @@ -26,11 +26,16 @@ template-ID = ID of molecule template specified in a separate molecule command hybrid args = list of one or more sub-styles, each with their args
accelerated styles (with same args): +
+Examples:
atom_style atomic atom_style bond atom_style full +atom_style full/cuda atom_style body nparticle 2 10 atom_style hybrid charge bond atom_style hybrid charge body nparticle 2 5 @@ -200,6 +205,31 @@ per-atom basis.LAMMPS can be extended with new atom styles as well as new body styles; see this section.
+
+ +Styles with a cuda or kk suffix are functionally the same as the +corresponding style without the suffix. They have been optimized to +run faster, depending on your available hardware, as discussed in +Section_accelerate of the manual. The +accelerated styles take the same arguments and should produce the same +results, except for round-off and precision issues. +
+Note that other acceleration packages in LAMMPS, specifically the GPU, +USER-OMP, and OPT packages do not use of accelerated atom styles. +
+These accelerated styles are part of the USER-CUDA and KOKKOS packages +respectively. They are only enabled if LAMMPS was built with those +packages. See the Making LAMMPS section +for more info. +
+You can specify the accelerated styles explicitly in your input script +by including their suffix, or you can use the -suffix command-line +switch when you invoke LAMMPS, or you can +use the suffix command in your input script. +
+See Section_accelerate of the manual for +more instructions on how to use the accelerated styles effectively. +
Restrictions:
This command cannot be used after the simulation box is defined by a diff --git a/doc/atom_style.txt b/doc/atom_style.txt index 1819324fd3..8690d30537 100644 --- a/doc/atom_style.txt +++ b/doc/atom_style.txt @@ -24,11 +24,16 @@ style = {angle} or {atomic} or {body} or {bond} or {charge} or {dipole} or \ template-ID = ID of molecule template specified in a separate "molecule"_molecule.html command {hybrid} args = list of one or more sub-styles, each with their args :pre +accelerated styles (with same args): + +style = {angle/cuda} or {atomic/cuda} or {atomic/kokkos} or {charge/cuda} or {full/cuda} :ul + [Examples:] atom_style atomic atom_style bond atom_style full +atom_style full/cuda atom_style body nparticle 2 10 atom_style hybrid charge bond atom_style hybrid charge body nparticle 2 5 @@ -196,6 +201,31 @@ per-atom basis. LAMMPS can be extended with new atom styles as well as new body styles; see "this section"_Section_modify.html. +:line + +Styles with a {cuda} or {kk} suffix are functionally the same as the +corresponding style without the suffix. They have been optimized to +run faster, depending on your available hardware, as discussed in +"Section_accelerate"_Section_accelerate.html of the manual. The +accelerated styles take the same arguments and should produce the same +results, except for round-off and precision issues. + +Note that other acceleration packages in LAMMPS, specifically the GPU, +USER-OMP, and OPT packages do not use of accelerated atom styles. + +These accelerated styles are part of the USER-CUDA and KOKKOS packages +respectively. They are only enabled if LAMMPS was built with those +packages. See the "Making LAMMPS"_Section_start.html#start_3 section +for more info. + +You can specify the accelerated styles explicitly in your input script +by including their suffix, or you can use the "-suffix command-line +switch"_Section_start.html#start_7 when you invoke LAMMPS, or you can +use the "suffix"_suffix.html command in your input script. + +See "Section_accelerate"_Section_accelerate.html of the manual for +more instructions on how to use the accelerated styles effectively. + [Restrictions:] This command cannot be used after the simulation box is defined by a diff --git a/doc/fix_nve.html b/doc/fix_nve.html index e70474fe02..e027559e82 100644 --- a/doc/fix_nve.html +++ b/doc/fix_nve.html @@ -13,6 +13,8 @@
fix nve/cuda command
+fix nve/kk command +
fix nve/omp command
Syntax: @@ -35,17 +37,18 @@ ensemble.
-Styles with a cuda, gpu, omp, or opt suffix are functionally -the same as the corresponding style without the suffix. They have -been optimized to run faster, depending on your available hardware, as -discussed in Section_accelerate of the -manual. The accelerated styles take the same arguments and should -produce the same results, except for round-off and precision issues. +
Styles with a cuda, gpu, kk, omp, or opt suffix are +functionally the same as the corresponding style without the suffix. +They have been optimized to run faster, depending on your available +hardware, as discussed in Section_accelerate +of the manual. The accelerated styles take the same arguments and +should produce the same results, except for round-off and precision +issues.
-These accelerated styles are part of the USER-CUDA, GPU, USER-OMP and OPT -packages, respectively. They are only enabled if LAMMPS was built with -those packages. See the Making LAMMPS -section for more info. +
These accelerated styles are part of the USER-CUDA, GPU, KOKKOS, +USER-OMP and OPT packages, respectively. They are only enabled if +LAMMPS was built with those packages. See the Making +LAMMPS section for more info.
You can specify the accelerated styles explicitly in your input script by including their suffix, or you can use the -suffix command-line diff --git a/doc/fix_nve.txt b/doc/fix_nve.txt index b43a78c628..46f842d371 100644 --- a/doc/fix_nve.txt +++ b/doc/fix_nve.txt @@ -8,6 +8,7 @@ fix nve command :h3 fix nve/cuda command :h3 +fix nve/kk command :h3 fix nve/omp command :h3 [Syntax:] @@ -30,17 +31,18 @@ ensemble. :line -Styles with a {cuda}, {gpu}, {omp}, or {opt} suffix are functionally -the same as the corresponding style without the suffix. They have -been optimized to run faster, depending on your available hardware, as -discussed in "Section_accelerate"_Section_accelerate.html of the -manual. The accelerated styles take the same arguments and should -produce the same results, except for round-off and precision issues. +Styles with a {cuda}, {gpu}, {kk}, {omp}, or {opt} suffix are +functionally the same as the corresponding style without the suffix. +They have been optimized to run faster, depending on your available +hardware, as discussed in "Section_accelerate"_Section_accelerate.html +of the manual. The accelerated styles take the same arguments and +should produce the same results, except for round-off and precision +issues. -These accelerated styles are part of the USER-CUDA, GPU, USER-OMP and OPT -packages, respectively. They are only enabled if LAMMPS was built with -those packages. See the "Making LAMMPS"_Section_start.html#start_3 -section for more info. +These accelerated styles are part of the USER-CUDA, GPU, KOKKOS, +USER-OMP and OPT packages, respectively. They are only enabled if +LAMMPS was built with those packages. See the "Making +LAMMPS"_Section_start.html#start_3 section for more info. You can specify the accelerated styles explicitly in your input script by including their suffix, or you can use the "-suffix command-line diff --git a/doc/fix_rigid.html b/doc/fix_rigid.html index 88ff880c02..aecf8bcfe8 100644 --- a/doc/fix_rigid.html +++ b/doc/fix_rigid.html @@ -764,10 +764,27 @@ of the run command. These fixes are not invoked during LAMMPS was built with that package. See the Making LAMMPS section for more info.
+Assigning a temperature via the velocity create +command to a system with rigid bodies may not have +the desired outcome for two reasons. First, the velocity command can +be invoked before the rigid-body fix is invoked or initialized and the +number of adjusted degrees of freedom (DOFs) is known. Thus it is not +possible to compute the target temperature correctly. Second, the +assigned velocities may be partially canceled when constraints are +first enforced, leading to a different temperature than desired. A +workaround for this is to perform a run 0 command, which +insures all DOFs are accounted for properly, and then rescale the +temperature to the desired value before performing a simulation. For +example: +
+velocity all create 300.0 12345 +run 0 # temperature may not be 300K +velocity all scale 300.0 # now it should be +Related commands:
delete_bonds, neigh_modify -exclude +exclude, fix shake
Default:
diff --git a/doc/fix_rigid.txt b/doc/fix_rigid.txt index 66a6c7bd67..3d0fc4afcc 100644 --- a/doc/fix_rigid.txt +++ b/doc/fix_rigid.txt @@ -746,10 +746,27 @@ These fixes are all part of the RIGID package. It is only enabled if LAMMPS was built with that package. See the "Making LAMMPS"_Section_start.html#start_3 section for more info. +Assigning a temperature via the "velocity create"_velocity.html +command to a system with "rigid bodies"_fix_rigid.html may not have +the desired outcome for two reasons. First, the velocity command can +be invoked before the rigid-body fix is invoked or initialized and the +number of adjusted degrees of freedom (DOFs) is known. Thus it is not +possible to compute the target temperature correctly. Second, the +assigned velocities may be partially canceled when constraints are +first enforced, leading to a different temperature than desired. A +workaround for this is to perform a "run 0"_run.html command, which +insures all DOFs are accounted for properly, and then rescale the +temperature to the desired value before performing a simulation. For +example: + +velocity all create 300.0 12345 +run 0 # temperature may not be 300K +velocity all scale 300.0 # now it should be :pre + [Related commands:] "delete_bonds"_delete_bonds.html, "neigh_modify"_neigh_modify.html -exclude +exclude, "fix shake"_fix_shake.html [Default:] diff --git a/doc/package.html b/doc/package.html index d707037dae..939fee6ff2 100644 --- a/doc/package.html +++ b/doc/package.html @@ -15,24 +15,11 @@package style args-