ICODE-ADC/lammps
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add support for HPE MPI to Kokkos package

Browse Source
This commit is contained in:
Stan Moore
2019-09-09 10:14:30 -06:00
parent cfd72aacce
commit 51786a459f
3 changed files with 21 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
doc/src/Run_options.txt
View File

@ -126,9 +126,10 @@ are intended for computational work like running LAMMPS. By default
Ng = 1 and Ns is not set. Ng = 1 and Ns is not set.
Depending on which flavor of MPI you are running, LAMMPS will look for Depending on which flavor of MPI you are running, LAMMPS will look for
one of these 3 environment variables one of these 4 environment variables
SLURM_LOCALID (various MPI variants compiled with SLURM support) SLURM_LOCALID (various MPI variants compiled with SLURM support)
MPT_LRANK (HPE MPI)
MV2_COMM_WORLD_LOCAL_RANK (Mvapich) MV2_COMM_WORLD_LOCAL_RANK (Mvapich)
OMPI_COMM_WORLD_LOCAL_RANK (OpenMPI) :pre OMPI_COMM_WORLD_LOCAL_RANK (OpenMPI) :pre

14
src/KOKKOS/kokkos.cpp
View File

@ -113,23 +113,37 @@ KokkosLMP::KokkosLMP(LAMMPS *lmp, int narg, char **arg) : Pointers(lmp)
} }
iarg += 2; iarg += 2;
int set_flag = 0;
char *str; char *str;
if ((str = getenv("SLURM_LOCALID"))) { if ((str = getenv("SLURM_LOCALID"))) {
int local_rank = atoi(str); int local_rank = atoi(str);
device = local_rank % ngpus; device = local_rank % ngpus;
if (device >= skip_gpu) device++; if (device >= skip_gpu) device++;
set_flag = 1;
}
if ((str = getenv("MPT_LRANK"))) {
int local_rank = atoi(str);
device = local_rank % ngpus;
if (device >= skip_gpu) device++;
set_flag = 1;
} }
if ((str = getenv("MV2_COMM_WORLD_LOCAL_RANK"))) { if ((str = getenv("MV2_COMM_WORLD_LOCAL_RANK"))) {
int local_rank = atoi(str); int local_rank = atoi(str);
device = local_rank % ngpus; device = local_rank % ngpus;
if (device >= skip_gpu) device++; if (device >= skip_gpu) device++;
set_flag = 1;
} }
if ((str = getenv("OMPI_COMM_WORLD_LOCAL_RANK"))) { if ((str = getenv("OMPI_COMM_WORLD_LOCAL_RANK"))) {
int local_rank = atoi(str); int local_rank = atoi(str);
device = local_rank % ngpus; device = local_rank % ngpus;
if (device >= skip_gpu) device++; if (device >= skip_gpu) device++;
set_flag = 1;
} }
if (ngpus > 1 && !set_flag)
error->all(FLERR,"Could not determine local MPI rank for multiple "
"GPUs with Kokkos CUDA because MPI library not recognized");
} else if (strcmp(arg[iarg],"t") == 0 || } else if (strcmp(arg[iarg],"t") == 0 ||
strcmp(arg[iarg],"threads") == 0) { strcmp(arg[iarg],"threads") == 0) {
nthreads = atoi(arg[iarg+1]); nthreads = atoi(arg[iarg+1]);

5
src/KOKKOS/kokkos.h
View File

@ -74,6 +74,11 @@ E: Invalid Kokkos command-line args
Self-explanatory. See Section 2.7 of the manual for details. Self-explanatory. See Section 2.7 of the manual for details.
E: Could not determine local MPI rank for multiple GPUs with Kokkos CUDA
because MPI library not recognized
The local MPI rank was not found in one of four supported environment variables.
E: GPUs are requested but Kokkos has not been compiled for CUDA E: GPUs are requested but Kokkos has not been compiled for CUDA
Recompile Kokkos with CUDA support to use GPUs. Recompile Kokkos with CUDA support to use GPUs.