ICODE-ADC/lammps
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update README for syntax and added arguments in run_tests.py

Browse Source
This commit is contained in:
Trung Nguyen
2024-09-06 12:46:42 -05:00
parent b2cc2582e1
commit 932f10e3b6
2 changed files with 92 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

118
tools/regression-tests/README
View File

@ -1,5 +1,5 @@
The script `run_tests.py` in this folder is used to perform regression tests
using in-place example scripts.
using in-place example scripts and provided log files as reference.
What this single script does is to launch the selected LAMMPS binary
using a testing configuration defined in a `.yaml` file (e.g., `config.yaml`)
@ -19,25 +19,43 @@ within the specified tolerances in the test configuration `config.yaml` file.
With the current features, users can:
+ specify which LAMMPS binary version to test (e.g., the version from a commit, or those from `lammps-testing`)
+ specify the examples subfolders (thus the reference log files) seperately (e.g. from other LAMMPS versions or commits)
+ specify tolerances for individual quantities for any input script to override the global values
+ launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffices)
+ skip certain input files if not interested, or no reference log file exists
+ simplify the main LAMMPS builds, as long as a LAMMPS binary is available
+ specify the examples subfolders (thus the reference log files) seperately (e.g. from other LAMMPS versions or commits), or
+ specify a file that lists of the examples input scripts to test
+ specify tolerances for individual quantities for any input script to override the global values in the config file
+ launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffixes)
+ skip certain input files (whose names match specified patterns) if not interested, or packaged not installed, or no reference log file exists
+ set a timeout for every input script run if they may take too long
+ skip numerical checks if the goal is just to check if the runs do not fail
Some benefits include:
+ separating regression testing from building LAMMPS
+ performing quick and full regression tests
+ keeping track of the testing progress to resume the testing from the last checkpoint (skipping completed runs)
+ distributing the input list across multiple processes by
splitting the list of input scripts into separate runs (there are ~800 input scripts under the top-level examples)
Input arguments:
+ the path to a LAMMPS binary (can be relative to the working directory)
+ a test configuration file (see tools/regression-tests/config.yaml for an example)
+ a text file that lists of folders where the input scripts reside and how many of them line by line, or
a text file that list of input scripts, or
the path to the top-level examples
Output:
+ failure.yaml : a dictionary of the failed runs and reasons
+ progress.yaml: full testing results of the tested input scripts with the status (completed, failed or skipped)
with error messages (for failed runs), and walltime (in seconds)
+ output.xml : testing results in the JUnit XML format
+ run.log : screen output and error of individual runs
Limitations:
- input scripts use thermo style multi (e.g., examples/peptide) do not work with the expected thermo output format
- input scripts that require partition runs (e.g. examples/neb) need a separate config file, e.g. "args: --partition 2x1"
- testing accelerator packages (GPU, INTEL, KOKKOS, OPENMP) need separate config files, "args: -sf omp -pk omp 4"
TODO:
+ keep track of the testing progress to resume the testing from the last checkpoint
+ distribute the input list across multiple processes via multiprocessing, or
split the list of input scripts into separate runs (there are 800+ input script under the top-level examples)
+ be able to be invoked from run_tests in the lammps-testing infrastruture
+ input scripts use thermo style multi (e.g., examples/peptide) do not work with the expected thermo output format
+ input scripts that require partition runs (e.g. examples/neb) need a separate config file, e.g., args: "--partition 3x1"
+ testing with accelerator packages (GPU, INTEL, KOKKOS, OPENMP) need separate config files, e.g., args: "-sf omp -pk omp 4"
The following Python packages need to be installed into an activated environment:
@ -45,40 +63,44 @@ The following Python packages need to be installed into an activated environment
source testing-env/bin/activate
pip install numpy pyyaml junit_xml
For all the supported arguments, run:
Example uses:
python3 tools/regression-tests/run_tests.py -h
Example uses (aka, tests for this script):
1) Simple use (using the provided tools/regression-tests/config.yaml and the examples/ folder at the top level)
python3 run_tests.py --lmp-bin=/path/to/lmp_binary
python3 run_tests.py --lmp-bin=build/lmp --config-file=tools/regression-tests/config.yaml
2) Use a custom testing configuration
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml
3) Specify a list of example folders
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml \
--example-folders="/path/to/examples/folder1;/path/to/examples/folder2"
--example-folders="/path/to/examples/melt;/path/to/examples/rigid"
The example folders can also be loaded from a text file list_subfolders1.txt:
The example subfolders can also be loaded from a text file list_subfolders1.txt:
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml \
--list-input=list_subfolders1.txt --output-file=output1.txt --progress-file=progress1.yaml \
--list-subfolders=list_subfolders1.txt --output-file=output1.txt --progress-file=progress1.yaml \
--log-file=run1.log
4) Test a LAMMPS binary with the whole top-level /examples folder in a LAMMPS source tree
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples
4) Specify a list of example input scripts (e.g. obtained from running tools/regression-tests/get-quick-list.py)
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml \
--list-input=input_list.txt
5) Analyze (dry run) the LAMMPS binary and whole top-level /examples folder in a LAMMPS source tree
5) Test a LAMMPS binary with the whole top-level /examples folder in a LAMMPS source tree
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples
--config-file=tools/regression-tests/config.yaml
6) Analyze the LAMMPS binary and whole top-level /examples folder in a LAMMPS source tree
and generate separate input lists for 8 workers:
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples \
--analyze --num-workers=8
This is used for splitting the subfolders into separate input lists and launching different instances
of run_tests.py simultaneously.
6) Prepare (dry run) for a quick regression test run that only runs inputs with commands and styles that
have changes in the current branch versus the selected upstream branch. Curb at 40 runs and split and
write out separate lists for 4 workers:
python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples \
--quick --quick-branch=origin/develop --quick-max= 40 --num-workers=4
The output of this run is 8 files folder-list-[0-7].txt that lists the subfolders
and 8 files input-list-[0-7].txt that lists the input scripts under the top-level example folders.
With these lists, one can launch multiple instances of run_tests.py simultaneously
each with a list of example subfolders (Case 3), or with a list of input scripts (Case 4).
An example of the test configuration `config.yaml` is given as below.
@ -113,17 +135,31 @@ An example of the test configuration `config.yaml` is given as below.
abs: 1e-2
rel: 1e-4
skip:
[ in.rigid.poems3,
in.rigid.poems4,
in.rigid.poems5,
[ in.displ,
in.displ2,
in.*_imd*,
]
nugget: 1.0
epsilon: 1e-16
timeout: 180
An example of the list of input scripts in a text file `list_subfolders1.txt`
An example of the list of example subfolders in a text file `list_subfolders1.txt`
/home/codes/lammps/examples/melt 1
/home/codes/lammps/examples/body 5
/home/codes/lammps/examples/PACKAGES/dielectric 2
/home/codes/lammps/examples/PACKAGES/tally 3
where the numbers are the number of input scripts (in.*) in the folders.
An example of the list of input scripts in a text file `input_list.txt`
/home/codes/lammps/examples/melt/in.melt
/home/codes/lammps/examples/body/in.body
/home/codes/lammps/examples/body/in.cubes
/home/codes/lammps/examples/PACKAGES/dielectric/in.confined
/home/codes/lammps/examples/PACKAGES/tally/in.pe
/home/codes/lammps/examples/PACKAGES/tally/in.force
/home/codes/lammps/examples/melt
/home/codes/lammps/examples/body
/home/codes/lammps/examples/PACKAGES/dielectric
/home/codes/lammps/examples/PACKAGES/tally

30
tools/regression-tests/run_tests.py
View File

@ -11,16 +11,19 @@ With the current features, users can:
+ specify the examples subfolders (thus the reference log files) seperately (e.g. from other LAMMPS versions or commits)
+ specify the list of examples input scripts to test
+ specify tolerances for individual quantities for any input script to override the global values
+ launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffices)
+ launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffixes)
+ skip certain input files (whose names match specified patterns) if not interested, or packaged not installed, or no reference log file exists
+ set a timeout for every input script run if they may take too long
+ skip numerical checks if the goal is just to check if the runs do not fail
+ simplify the main LAMMPS builds, as long as a LAMMPS binary is available
+ keep track of the testing progress to resume the testing from the last checkpoint (skipping completed runs)
+ distribute the input list across multiple processes via multiprocessing, or
split the list of input scripts into separate runs (there are 800+ input script under the top-level examples)
Some benefits include:
+ separating regression testing from building LAMMPS
+ performing quick and full regression tests
+ keeping track of the testing progress to resume the testing from the last checkpoint (skipping completed runs)
+ distributing the input list across multiple processes by
splitting the list of input scripts into separate runs (there are ~800 input scripts under the top-level examples)
Input arguments:
+ the path to a LAMMPS binary (can be relative to the working directory)
+ a test configuration file (see tools/regression-tests/config.yaml for an example)
@ -40,9 +43,6 @@ Limitations:
- input scripts that require partition runs (e.g. examples/neb) need a separate config file, e.g. args: "--partition 3x1"
- testing accelerator packages (GPU, INTEL, KOKKOS, OPENMP) need separate config files, "args: -sf omp -pk omp 4"
TODO:
+ be able to be invoked from run_tests in the lammps-testing infrastruture
The following Python packages need to be installed into an activated environment:
python3 -m venv testing-env
@ -1049,12 +1049,6 @@ if __name__ == "__main__":
parser.add_argument("--list-input", dest="list_input", default="", help="File that lists the input scripts")
parser.add_argument("--list-subfolders", dest="list_subfolders", default="", help="File that lists the subfolders")
parser.add_argument("--num-workers", dest="num_workers", default=1, help="Number of workers")
parser.add_argument("--gen-ref",dest="genref", action='store_true', default=False,
help="Generating reference data")
parser.add_argument("--verbose",dest="verbose", action='store_true', default=False,
help="Verbose output")
parser.add_argument("--resume",dest="resume", action='store_true', default=False,
help="Resume the test run")
parser.add_argument("--output-file",dest="output", default=output_file, help="Output file")
parser.add_argument("--log-file",dest="logfile", default=log_file, help="Log file")
parser.add_argument("--progress-file",dest="progress_file", default=progress_file, help="Progress file")
@ -1069,7 +1063,13 @@ if __name__ == "__main__":
parser.add_argument("--quick-max", dest="quick_max", default=50,
help="Maximum number of inputs to randomly select")
parser.add_argument("--skip-numerical-check",dest="skip_numerical_check", action='store_true', default=False,
help="Generating reference data")
help="Skip numerical checks")
parser.add_argument("--gen-ref",dest="genref", action='store_true', default=False,
help="Generating reference log files")
parser.add_argument("--verbose",dest="verbose", action='store_true', default=False,
help="Verbose screen output")
parser.add_argument("--resume",dest="resume", action='store_true', default=False,
help="Resume the test run from the list of inputs given the progress in progress.yaml")
args = parser.parse_args()