2

tuning

Mark Olesen edited this page 2023-12-22 13:07:25 +01:00

Table of Contents

• Changing switch values
• Making central changes
• Making case-specific changes
• Making job-specific changes
• Parallel Tuning
• Comments on parallel tuning

The OpenFOAM etc/controlDict file contains a list of commonly used user-configurable switches, which are divided into three broad categories:

• InfoSwitches : controls output and some system settings
• OptimisationSwitches : various tuning parameters
• DebugSwitches : enable low-level debugging output (mostly for developers or deeper diagnostics when things go wrong)

In most cases the individual switch entries are registered internally by OpenFOAM, which means that they can also be modified by user input.

[[TOC]]

Changing switch values

Making central changes

If changes are needed on a site-wide basis, the OpenFOAM etc/controlDict can be edited directly. This ensures that changes will be used for all jobs and users. However, these changes must be consistent across all compute nodes otherwise the effect is unpredicable. Additionally, changes to the etc/controlDict will be lost when changing to a new version.

Making case-specific changes

To make case-specific changes, simply add a corresponding sub-dictionary within the case system/controlDict. For example,

OptimisationSwitches
{
    // Force dumping (at next timestep) upon signal: 12 = USR2
    writeNowSignal  12;
}

Making job-specific changes

(largely v1912 and later)
The different switches can also be adjusted via corresponding command-line options (-info-switch, -opt-switch, -debug-switch). The switch can be specified without a value, in which case a default value is assumed (typically "1", since many switches are integers), or with a specified value.

Some examples:

$ simpleFoam -info-switch writeOptionalEntries -dry-run

$ foamDictionary -debug-switch fieldExpr=4 -expand mydict

$ simpleFoam -opt-switch maxMasterFileBufferSize=1e6

Parallel Tuning

(largely v2306 and later)
As part of the exaFOAM project a variety of communication bottlenecks have been identified and avoided. Some of the different algorithms and settings have been made configurable via OptimisationSwitches (MPI/Parallel settings).

One of the algorithmic changes is to use NBX (non-blocking exchange) instead of an all-to-all when exchanging buffer sizes, which should improve scaling above 2k ranks and have no disadvantage with lower numbers of ranks. Since these settings are still under testing, they are disable by default but can be selected via OptimisationSwitches.

Some of the OptimisationSwitches relevant for MPI/Parallel:

• nbx.min NBX min size (default: 0)
  The minimum number of processor ranks to use the non-blocking exchange (NBX) algorithm when exchanging sizes. For values less than 1, an all-to-all is used instead.
• nbx.tuning NBX tuning parameter (default: 0)
  For some systems an initial barrier may be necessary for NBX to avoid race conditions. A value of 1 or greater can currently be used to enable an initial barrier.
• pbufs.tuning PstreamBuffers tuning (default: -1)
  • value = 0 (legacy PEX) : use all-to-all to exchange buffer sizes, followed by point-to-point communication for the contents.
  • value = 1 (hybrid PEX) : use NBX to exchange buffer sizes, followed by point-to-point communication for the contents.
• nPollProcInterfaces processor patch polling (default: 0)
  
  • value = 0 (default) : non-polling. Wait for all exchanges to complete before dispatching.
  • value = -1 : wait for any requests to finish and dispatch when possible followed by point-to-point communication for the contents.
  • value >= 1 : deprecated (will be removed)

Caution in OpenFOAM-v2306 the pbufs.tuning used slightly different values (-1 : legacy, 0 : hybrid).

Comments on parallel tuning

It is expected that large simulations with many geometry changes will be the primary beneficiary of the NBX algorithm since PstreamBuffers is used there. The use of NBX can be selected separately for PstreamBuffers and for regular size exchanges.

The full benefits of polling processor interfaces (nPollProcInterfaces) remain unclear.

---

Copyright (C) 2023 OpenCFD Ltd.