- the readList<T>(Istream&) function was introduced to handle command
-options with either a single or a list value, but was also used for
the #remove dictionary directive. However, the parsing was fragile
if the list did not start with a '('.
Now handle command-line arg/option list directly (via ITstream)
and #remove with special-purpose reading of a string or word list.
This removes ambiguity and reduces potential future problems.
STYLE: use ITstream instead of IStringStream for command-line lookups
- parses directly to a tokenList without a string copy.
- If the entry could be directly inserted: a pointer to the inserted entry.
- If a dictionary merge was required: a pointer to the dictionary that
received the entry.
- Return nullptr on any type of insertion failure.
This change is code compatible with existing code since it only alters
a bool return value to be a pointer return value.
- Instead of relying on #inputMode to effect a global change it is now
possible (and recommended) to a temporary change in the inputMode
for the following entry.
#default : provide default value if entry is not already defined
#overwrite : silently remove a previously existing entry
#warn : warn about duplicate entries
#error : error if any duplicate entries occur
#merge : merge sub-dictionaries when possible (the default mode)
This is generally less cumbersome than the switching the global
inputMode. For example to provide a set of fallback values.
#includeIfPresent "user-files"
...
#default value uniform 10;
vs.
#includeIfPresent "user-files"
#inputMode protect
...
value uniform 10;
#inputMode merge // _Assuming_ we actually had this before
These directives can also be used to suppress the normal dictionary
merge semantics:
#overwrite dict { entry val; ... }
- patterns only supported for the final element.
To create an element as a pattern instead of a word, an embedded
string quote (single or double) is used for that element.
Any of the following examples:
"/top/sub/dict/'(p|U).*" 100;
"/top/sub/dict/'(p|U).*'" 100;
"/top/sub/dict/\"(p|U).*" 100;
"/top/sub/dict/\"(p|U).*\"" 100;
are equivalent to the longer form:
top
{
sub
{
dict
{
"(p|U).*" 100;
}
}
}
It is not currently possible to auto-vivify intermediate
dictionaries with patterns.
NOK "/nonexistent.*/value" 100;
OK "/existing.*/value" 100;
- full scoping also works for the #remove directive
#remove "/dict1/subdict2/entry1"
Original commit message:
------------------------
Parallel IO: New collated file format
When an OpenFOAM simulation runs in parallel, the data for decomposed fields and
mesh(es) has historically been stored in multiple files within separate
directories for each processor. Processor directories are named 'processorN',
where N is the processor number.
This commit introduces an alternative "collated" file format where the data for
each decomposed field (and mesh) is collated into a single file, which is
written and read on the master processor. The files are stored in a single
directory named 'processors'.
The new format produces significantly fewer files - one per field, instead of N
per field. For large parallel cases, this avoids the restriction on the number
of open files imposed by the operating system limits.
The file writing can be threaded allowing the simulation to continue running
while the data is being written to file. NFS (Network File System) is not
needed when using the the collated format and additionally, there is an option
to run without NFS with the original uncollated approach, known as
"masterUncollated".
The controls for the file handling are in the OptimisationSwitches of
etc/controlDict:
OptimisationSwitches
{
...
//- Parallel IO file handler
// uncollated (default), collated or masterUncollated
fileHandler uncollated;
//- collated: thread buffer size for queued file writes.
// If set to 0 or not sufficient for the file size threading is not used.
// Default: 2e9
maxThreadFileBufferSize 2e9;
//- masterUncollated: non-blocking buffer size.
// If the file exceeds this buffer size scheduled transfer is used.
// Default: 2e9
maxMasterFileBufferSize 2e9;
}
When using the collated file handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory in bytes that
is allocated. If the data exceeds this size, the write does not use threading.
When using the masterUncollated file handling, non-blocking MPI communication
requires a sufficiently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size in bytes of the buffer. If the
data exceeds this size, the system uses scheduled communication.
The installation defaults for the fileHandler choice, maxThreadFileBufferSize
and maxMasterFileBufferSize (set in etc/controlDict) can be over-ridden within
the case controlDict file, like other parameters. Additionally the fileHandler
can be set by:
- the "-fileHandler" command line argument;
- a FOAM_FILEHANDLER environment variable.
A foamFormatConvert utility allows users to convert files between the collated
and uncollated formats, e.g.
mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated
An example case demonstrating the file handling methods is provided in:
$FOAM_TUTORIALS/IO/fileHandling
The work was undertaken by Mattijs Janssens, in collaboration with Henry Weller.
- The logic for switching input-mode was previously completely
encapsulated within the #inputMode directive, but without any
programming equivalent. Furthermore, the encapsulation in inputMode
made the logic less clear in other places.
Exposing the inputMode as an enum with direct access from entry
simplifies things a fair bit.
- eliminate one level of else/if nesting in entryIO.C for clearer logic
- for dictionary function entries, simply use
addNamedToMemberFunctionSelectionTable() and avoid defining a type()
as a static. For most function entries the information is only used
to get a name for the selection table lookup anyhow.
- use allocator class to wrap the stream pointers instead of passing
them into ISstream, OSstream and using a dynamic cast to delete
then. This is especially important if we will have a bidirectional
stream (can't delete twice!).
STYLE:
- file stream constructors with std::string (C++11)
- for rewind, explicit about in|out direction. This is not currently
important, but avoids surprises with any future bidirectional access.
- combined string streams in StringStream.H header.
Similar to <sstream> include that has both input and output string
streams.
- ensure proper and sensible handling of empty names.
Eg, isDir(""), isFile("") are no-ops, and avoid file-stat
- rmDir:
* optional 'silent' option to suppress messages.
* removes all possible sub-entries, instead of just giving up on
the first problem encountered.
- reduced code duplication in etcFiles
ENH: provide WM_USER_RESOURCE_DIRNAME define (in foamVersion.H)
- this is still a hard-coded value, but at least centrally available
- provides support for manipulating polyMesh/boundary
- changed behaviour of disableFunctionEntries option to preserve
#include
- dictionary: added reading of lists of dictionaries.
+ each list element may be accessed using the 'entryDDD' keyword
according to their list index.
Patch contributed by Mattijs Janssens
e.g.
functions
{
#includeFunc mag(U)
}
executes 'mag' on the field 'U' writing the field 'mag(U)'.
The equivalent post-processing command is
postProcess -func 'mag(U)'
with the more general and flexible 'postProcess' utility and '-postProcess' solver option
Rationale
---------
Both the 'postProcess' utility and '-postProcess' solver option use the
same extensive set of functionObjects available for data-processing
during the run avoiding the substantial code duplication necessary for
the 'foamCalc' and 'postCalc' utilities and simplifying maintenance.
Additionally consistency is guaranteed between solver data processing
and post-processing.
The functionObjects have been substantially re-written and generalized
to simplify development and encourage contribution.
Configuration
-------------
An extensive set of simple functionObject configuration files are
provided in
OpenFOAM-dev/etc/caseDicts/postProcessing
and more will be added in the future. These can either be copied into
'<case>/system' directory and included into the 'controlDict.functions'
sub-dictionary or included directly from 'etc/caseDicts/postProcessing'
using the '#includeEtc' directive or the new and more convenient
'#includeFunc' directive which searches the
'<etc>/caseDicts/postProcessing' directories for the selected
functionObject, e.g.
functions
{
#includeFunc Q
#includeFunc Lambda2
}
'#includeFunc' first searches the '<case>/system' directory in case
there is a local configuration.
Description of #includeFunc
---------------------------
Specify a functionObject dictionary file to include, expects the
functionObject name to follow (without quotes).
Search for functionObject dictionary file in
user/group/shipped directories.
The search scheme allows for version-specific and
version-independent files using the following hierarchy:
- \b user settings:
- ~/.OpenFOAM/\<VERSION\>/caseDicts/postProcessing
- ~/.OpenFOAM/caseDicts/postProcessing
- \b group (site) settings (when $WM_PROJECT_SITE is set):
- $WM_PROJECT_SITE/\<VERSION\>/caseDicts/postProcessing
- $WM_PROJECT_SITE/caseDicts/postProcessing
- \b group (site) settings (when $WM_PROJECT_SITE is not set):
- $WM_PROJECT_INST_DIR/site/\<VERSION\>/caseDicts/postProcessing
- $WM_PROJECT_INST_DIR/site/caseDicts/postProcessing
- \b other (shipped) settings:
- $WM_PROJECT_DIR/etc/caseDicts/postProcessing
An example of the \c \#includeFunc directive:
\verbatim
#includeFunc <funcName>
\endverbatim
postProcess
-----------
The 'postProcess' utility and '-postProcess' solver option provide the
same set of controls to execute functionObjects after the run either by
reading a specified set of fields to process in the case of
'postProcess' or by reading all fields and models required to start the
run in the case of '-postProcess' for each selected time:
postProcess -help
Usage: postProcess [OPTIONS]
options:
-case <dir> specify alternate case directory, default is the cwd
-constant include the 'constant/' dir in the times list
-dict <file> read control dictionary from specified location
-field <name> specify the name of the field to be processed, e.g. U
-fields <list> specify a list of fields to be processed, e.g. '(U T p)' -
regular expressions not currently supported
-func <name> specify the name of the functionObject to execute, e.g. Q
-funcs <list> specify the names of the functionObjects to execute, e.g.
'(Q div(U))'
-latestTime select the latest time
-newTimes select the new times
-noFunctionObjects
do not execute functionObjects
-noZero exclude the '0/' dir from the times list, has precedence
over the -withZero option
-parallel run in parallel
-region <name> specify alternative mesh region
-roots <(dir1 .. dirN)>
slave root directories for distributed running
-time <ranges> comma-separated time ranges - eg, ':10,20,40:70,1000:'
-srcDoc display source code in browser
-doc display application documentation in browser
-help print the usage
pimpleFoam -postProcess -help
Usage: pimpleFoam [OPTIONS]
options:
-case <dir> specify alternate case directory, default is the cwd
-constant include the 'constant/' dir in the times list
-dict <file> read control dictionary from specified location
-field <name> specify the name of the field to be processed, e.g. U
-fields <list> specify a list of fields to be processed, e.g. '(U T p)' -
regular expressions not currently supported
-func <name> specify the name of the functionObject to execute, e.g. Q
-funcs <list> specify the names of the functionObjects to execute, e.g.
'(Q div(U))'
-latestTime select the latest time
-newTimes select the new times
-noFunctionObjects
do not execute functionObjects
-noZero exclude the '0/' dir from the times list, has precedence
over the -withZero option
-parallel run in parallel
-postProcess Execute functionObjects only
-region <name> specify alternative mesh region
-roots <(dir1 .. dirN)>
slave root directories for distributed running
-time <ranges> comma-separated time ranges - eg, ':10,20,40:70,1000:'
-srcDoc display source code in browser
-doc display application documentation in browser
-help print the usage
The functionObjects to execute may be specified on the command-line
using the '-func' option for a single functionObject or '-funcs' for a
list, e.g.
postProcess -func Q
postProcess -funcs '(div(U) div(phi))'
In the case of 'Q' the default field to process is 'U' which is
specified in and read from the configuration file but this may be
overridden thus:
postProcess -func 'Q(Ua)'
as is done in the example above to calculate the two forms of the divergence of
the velocity field. Additional fields which the functionObjects may depend on
can be specified using the '-field' or '-fields' options.
The 'postProcess' utility can only be used to execute functionObjects which
process fields present in the time directories. However, functionObjects which
depend on fields obtained from models, e.g. properties derived from turbulence
models can be executed using the '-postProcess' of the appropriate solver, e.g.
pisoFoam -postProcess -func PecletNo
or
sonicFoam -postProcess -func MachNo
In this case all required fields will have already been read so the '-field' or
'-fields' options are not be needed.
Henry G. Weller
CFD Direct Ltd.
Description
Specify an etc file to include when reading dictionaries, expects a
single string to follow.
Searches for files from user/group/shipped directories.
The search scheme allows for version-specific and
version-independent files using the following hierarchy:
- \b user settings:
- ~/.OpenFOAM/\<VERSION\>
- ~/.OpenFOAM/
- \b group (site) settings (when $WM_PROJECT_SITE is set):
- $WM_PROJECT_SITE/\<VERSION\>
- $WM_PROJECT_SITE
- \b group (site) settings (when $WM_PROJECT_SITE is not set):
- $WM_PROJECT_INST_DIR/site/\<VERSION\>
- $WM_PROJECT_INST_DIR/site/
- \b other (shipped) settings:
- $WM_PROJECT_DIR/etc/
An example of the \c \#includeEtc directive:
\verbatim
#includeEtc "etcFile"
\endverbatim
The usual expansion of environment variables and other constructs is
retained.
When using #include or #includeIfPresent, it is easy to generate an
empty file name. For example,
#include "~OpenFOAM/missing-path/file"
On error, findEtcFile() returns an empty fileName. This was treated as
a relative name, which meant there was an attempt to open the parent directory
as a file. The resulting error message was confusing:
--> FOAM FATAL IO ERROR:
Attempt to put back onto bad stream
file: /CASE/PATH/system at line 1.
From function void Istream::putBack(const token&)
in file db/IOstreams/IOstreams/Istream.C at line 34.
- fix by leaving empty expansions as-is
