update README for syntax and added arguments in run_tests.py

tools/regression-tests/README

@@ -1,5 +1,5 @@
 The script `run_tests.py` in this folder is used to perform regression tests
-using in-place example scripts.
+using in-place example scripts and provided log files as reference.
 
 What this single script does is to launch the selected LAMMPS binary
 using a testing configuration defined in a `.yaml` file (e.g., `config.yaml`)
@@ -19,25 +19,43 @@ within the specified tolerances in the test configuration `config.yaml` file.
 With the current features, users can:
 
     + specify which LAMMPS binary version to test (e.g., the version from a commit, or those from `lammps-testing`)
-    + specify the examples subfolders (thus the reference log files) seperately (e.g. from other LAMMPS versions or commits)
+    + specify the examples subfolders (thus the reference log files) seperately (e.g. from other LAMMPS versions or commits), or
-    + specify tolerances for individual quantities for any input script to override the global values
+    + specify a file that lists of the examples input scripts to test
-    + launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffices)
+    + specify tolerances for individual quantities for any input script to override the global values in the config file
-    + skip certain input files if not interested, or no reference log file exists
+    + launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffixes)
-    + simplify the main LAMMPS builds, as long as a LAMMPS binary is available
+    + skip certain input files (whose names match specified patterns) if not interested, or packaged not installed, or no reference log file exists
+    + set a timeout for every input script run if they may take too long
+    + skip numerical checks if the goal is just to check if the runs do not fail
+
+Some benefits include:
+
+    + separating regression testing from building LAMMPS
+    + performing quick and full regression tests
+    + keeping track of the testing progress to resume the testing from the last checkpoint (skipping completed runs)
+    + distributing the input list across multiple processes by
+      splitting the list of input scripts into separate runs (there are ~800 input scripts under the top-level examples)
+
+Input arguments:
+
+    + the path to a LAMMPS binary (can be relative to the working directory)
+    + a test configuration file (see tools/regression-tests/config.yaml for an example)
+    + a text file that lists of folders where the input scripts reside and how many of them line by line, or
+      a text file that list of input scripts, or
+      the path to the top-level examples
+
+Output:
+
+    + failure.yaml : a dictionary of the failed runs and reasons
+    + progress.yaml: full testing results of the tested input scripts with the status (completed, failed or skipped)
+                     with error messages (for failed runs), and walltime (in seconds)
+    + output.xml   :    testing results in the JUnit XML format
+    + run.log      :       screen output and error of individual runs
 
 Limitations:
 
-    - input scripts use thermo style multi (e.g., examples/peptide) do not work with the expected thermo output format
+    + input scripts use thermo style multi (e.g., examples/peptide) do not work with the expected thermo output format
-    - input scripts that require partition runs (e.g. examples/neb) need a separate config file, e.g. "args: --partition 2x1"
+    + input scripts that require partition runs (e.g. examples/neb) need a separate config file, e.g., args: "--partition 3x1"
-    - testing accelerator packages (GPU, INTEL, KOKKOS, OPENMP) need separate config files, "args: -sf omp -pk omp 4"
+    + testing with accelerator packages (GPU, INTEL, KOKKOS, OPENMP) need separate config files, e.g., args: "-sf omp -pk omp 4"
-
-TODO:
-
-    + keep track of the testing progress to resume the testing from the last checkpoint
-    + distribute the input list across multiple processes via multiprocessing, or
-      split the list of input scripts into separate runs (there are 800+ input script under the top-level examples)
-    + be able to be invoked from run_tests in the lammps-testing infrastruture
-
 
 The following Python packages need to be installed into an activated environment:
 
@@ -45,40 +63,44 @@ The following Python packages need to be installed into an activated environment
     source testing-env/bin/activate
     pip install numpy pyyaml junit_xml
 
+For all the supported arguments, run:
 
-Example uses:
+    python3 tools/regression-tests/run_tests.py -h
+
+Example uses (aka, tests for this script):
 
     1) Simple use (using the provided tools/regression-tests/config.yaml and the examples/ folder at the top level)
-           python3 run_tests.py --lmp-bin=/path/to/lmp_binary
+           python3 run_tests.py --lmp-bin=build/lmp --config-file=tools/regression-tests/config.yaml
 
     2) Use a custom testing configuration
            python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml
 
     3) Specify a list of example folders
            python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml \
-                --example-folders="/path/to/examples/folder1;/path/to/examples/folder2"
+                --example-folders="/path/to/examples/melt;/path/to/examples/rigid"
 
-       The example folders can also be loaded from a text file list_subfolders1.txt:
+       The example subfolders can also be loaded from a text file list_subfolders1.txt:
            python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml \
-                --list-input=list_subfolders1.txt --output-file=output1.txt --progress-file=progress1.yaml \
+                --list-subfolders=list_subfolders1.txt --output-file=output1.txt --progress-file=progress1.yaml \
                 --log-file=run1.log
 
-    4) Test a LAMMPS binary with the whole top-level /examples folder in a LAMMPS source tree
+    4) Specify a list of example input scripts (e.g. obtained from running tools/regression-tests/get-quick-list.py)
-           python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples
+            python3 run_tests.py --lmp-bin=/path/to/lmp_binary --config-file=/path/to/config/file/config.yaml \
+                --list-input=input_list.txt
 
-    5) Analyze (dry run) the LAMMPS binary and whole top-level /examples folder in a LAMMPS source tree
+    5) Test a LAMMPS binary with the whole top-level /examples folder in a LAMMPS source tree
+           python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples
+                --config-file=tools/regression-tests/config.yaml
+
+    6) Analyze the LAMMPS binary and whole top-level /examples folder in a LAMMPS source tree
        and generate separate input lists for 8 workers:
            python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples \
                 --analyze --num-workers=8
 
-       This is used for splitting the subfolders into separate input lists and launching different instances
+       The output of this run is 8 files folder-list-[0-7].txt that lists the subfolders
-       of run_tests.py simultaneously.
+       and 8 files input-list-[0-7].txt that lists the input scripts under the top-level example folders.
-
+       With these lists, one can launch multiple instances of run_tests.py simultaneously
-    6) Prepare (dry run) for a quick regression test run that only runs inputs with commands and styles that
+       each with a list of example subfolders (Case 3), or with a list of input scripts (Case 4).
-       have changes in the current branch versus the selected upstream branch. Curb at 40 runs and split and
-       write out separate lists for 4 workers:
-           python3 run_tests.py --lmp-bin=/path/to/lmp_binary --examples-top-level=/path/to/lammps/examples \
-                --quick --quick-branch=origin/develop --quick-max= 40 --num-workers=4
 
 An example of the test configuration `config.yaml` is given as below.
 
@@ -113,17 +135,31 @@ An example of the test configuration `config.yaml` is given as below.
           abs: 1e-2
           rel: 1e-4
     skip:
-      [ in.rigid.poems3,
+      [ in.displ,
-        in.rigid.poems4,
+        in.displ2,
-        in.rigid.poems5,
+        in.*_imd*,
       ]
 
     nugget: 1.0
     epsilon: 1e-16
+    timeout: 180
 
-An example of the list of input scripts in a text file `list_subfolders1.txt`
+An example of the list of example subfolders in a text file `list_subfolders1.txt`
+
+    /home/codes/lammps/examples/melt 1
+    /home/codes/lammps/examples/body 5
+    /home/codes/lammps/examples/PACKAGES/dielectric 2
+    /home/codes/lammps/examples/PACKAGES/tally 3
+
+where the numbers are the number of input scripts (in.*) in the folders.
+
+
+An example of the list of input scripts in a text file `input_list.txt`
+
+    /home/codes/lammps/examples/melt/in.melt
+    /home/codes/lammps/examples/body/in.body
+    /home/codes/lammps/examples/body/in.cubes
+    /home/codes/lammps/examples/PACKAGES/dielectric/in.confined
+    /home/codes/lammps/examples/PACKAGES/tally/in.pe
+    /home/codes/lammps/examples/PACKAGES/tally/in.force
 
-/home/codes/lammps/examples/melt
-/home/codes/lammps/examples/body
-/home/codes/lammps/examples/PACKAGES/dielectric
-/home/codes/lammps/examples/PACKAGES/tally

tools/regression-tests/run_tests.py

@@ -11,15 +11,18 @@ With the current features, users can:
     + specify the examples subfolders (thus the reference log files) seperately (e.g. from other LAMMPS versions or commits)
     + specify the list of examples input scripts to test
     + specify tolerances for individual quantities for any input script to override the global values
-    + launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffices)
+    + launch tests with `mpirun` with all supported command line features (multiple procs, multiple paritions, and suffixes)
     + skip certain input files (whose names match specified patterns) if not interested, or packaged not installed, or no reference log file exists
     + set a timeout for every input script run if they may take too long
     + skip numerical checks if the goal is just to check if the runs do not fail
-    + simplify the main LAMMPS builds, as long as a LAMMPS binary is available
-    + keep track of the testing progress to resume the testing from the last checkpoint (skipping completed runs)
-    + distribute the input list across multiple processes via multiprocessing, or
-      split the list of input scripts into separate runs (there are 800+ input script under the top-level examples)
 
+Some benefits include:
+
+    + separating regression testing from building LAMMPS
+    + performing quick and full regression tests
+    + keeping track of the testing progress to resume the testing from the last checkpoint (skipping completed runs)
+    + distributing the input list across multiple processes by
+      splitting the list of input scripts into separate runs (there are ~800 input scripts under the top-level examples)
       
 Input arguments:
     + the path to a LAMMPS binary (can be relative to the working directory)
@@ -40,9 +43,6 @@ Limitations:
     - input scripts that require partition runs (e.g. examples/neb) need a separate config file, e.g. args: "--partition 3x1"
     - testing accelerator packages (GPU, INTEL, KOKKOS, OPENMP) need separate config files, "args: -sf omp -pk omp 4"
 
-TODO:
-    + be able to be invoked from run_tests in the lammps-testing infrastruture
-
 The following Python packages need to be installed into an activated environment:
 
     python3 -m venv testing-env
@@ -1049,12 +1049,6 @@ if __name__ == "__main__":
     parser.add_argument("--list-input", dest="list_input", default="", help="File that lists the input scripts")
     parser.add_argument("--list-subfolders", dest="list_subfolders", default="", help="File that lists the subfolders")
     parser.add_argument("--num-workers", dest="num_workers", default=1, help="Number of workers")
-    parser.add_argument("--gen-ref",dest="genref", action='store_true', default=False,
-                        help="Generating reference data")
-    parser.add_argument("--verbose",dest="verbose", action='store_true', default=False,
-                        help="Verbose output")
-    parser.add_argument("--resume",dest="resume", action='store_true', default=False,
-                        help="Resume the test run")
     parser.add_argument("--output-file",dest="output", default=output_file, help="Output file")
     parser.add_argument("--log-file",dest="logfile", default=log_file, help="Log file")
     parser.add_argument("--progress-file",dest="progress_file", default=progress_file, help="Progress file")
@@ -1069,7 +1063,13 @@ if __name__ == "__main__":
     parser.add_argument("--quick-max", dest="quick_max", default=50,
                         help="Maximum number of inputs to randomly select")
     parser.add_argument("--skip-numerical-check",dest="skip_numerical_check", action='store_true', default=False,
-                        help="Generating reference data")
+                        help="Skip numerical checks")
+    parser.add_argument("--gen-ref",dest="genref", action='store_true', default=False,
+                        help="Generating reference log files")
+    parser.add_argument("--verbose",dest="verbose", action='store_true', default=False,
+                        help="Verbose screen output")
+    parser.add_argument("--resume",dest="resume", action='store_true', default=False,
+                        help="Resume the test run from the list of inputs given the progress in progress.yaml")
 
     args = parser.parse_args()