- a failed attempt to read a value (eg, word, label, scalar) from a
stream now always provokes a FatalIOError.
This helps avoid some difficult to trace input errors.
- In addition to the traditional Flex-based parser, added a Ragel-based
parser and a handwritten one.
Some representative timings for reading 5874387 points (1958129 tris):
Flex Ragel Manual
5.2s 4.8s 6.7s total reading time
3.8s 3.4s 5.3s without point merging
- more consistent with STL practices for function classes.
- string::hash function class now operates on std::string rather
than Foam::string since we have now avoided inadvertent use of
string conversion from int in more places.
Original commit message:
------------------------
Parallel IO: New collated file format
When an OpenFOAM simulation runs in parallel, the data for decomposed fields and
mesh(es) has historically been stored in multiple files within separate
directories for each processor. Processor directories are named 'processorN',
where N is the processor number.
This commit introduces an alternative "collated" file format where the data for
each decomposed field (and mesh) is collated into a single file, which is
written and read on the master processor. The files are stored in a single
directory named 'processors'.
The new format produces significantly fewer files - one per field, instead of N
per field. For large parallel cases, this avoids the restriction on the number
of open files imposed by the operating system limits.
The file writing can be threaded allowing the simulation to continue running
while the data is being written to file. NFS (Network File System) is not
needed when using the the collated format and additionally, there is an option
to run without NFS with the original uncollated approach, known as
"masterUncollated".
The controls for the file handling are in the OptimisationSwitches of
etc/controlDict:
OptimisationSwitches
{
...
//- Parallel IO file handler
// uncollated (default), collated or masterUncollated
fileHandler uncollated;
//- collated: thread buffer size for queued file writes.
// If set to 0 or not sufficient for the file size threading is not used.
// Default: 2e9
maxThreadFileBufferSize 2e9;
//- masterUncollated: non-blocking buffer size.
// If the file exceeds this buffer size scheduled transfer is used.
// Default: 2e9
maxMasterFileBufferSize 2e9;
}
When using the collated file handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory in bytes that
is allocated. If the data exceeds this size, the write does not use threading.
When using the masterUncollated file handling, non-blocking MPI communication
requires a sufficiently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size in bytes of the buffer. If the
data exceeds this size, the system uses scheduled communication.
The installation defaults for the fileHandler choice, maxThreadFileBufferSize
and maxMasterFileBufferSize (set in etc/controlDict) can be over-ridden within
the case controlDict file, like other parameters. Additionally the fileHandler
can be set by:
- the "-fileHandler" command line argument;
- a FOAM_FILEHANDLER environment variable.
A foamFormatConvert utility allows users to convert files between the collated
and uncollated formats, e.g.
mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated
An example case demonstrating the file handling methods is provided in:
$FOAM_TUTORIALS/IO/fileHandling
The work was undertaken by Mattijs Janssens, in collaboration with Henry Weller.
- consolidate word::validated() into word::validate() and also allow
as short form for string::validate<word>(). Also less confusing than
having similarly named methods that essentially do the same thing.
- more consistent const access when iterating over strings
- add valid(char) for keyType and wordRe
- ensure that the string-related classes have consistently similar
matching methods. Use operator()(const std::string) as an entry
point for the match() method, which makes it easier to use for
filters and predicates. In some cases this will also permit using
a HashSet as a match predicate.
regExp
====
- the set method now returns a bool to signal that the requested
pattern was compiled.
wordRe
====
- have separate constructors with the compilation option (was previously
a default parameter). This leaves the single parameter constructor
explicit, but the two parameter version is now non-explicit, which
makes it easier to use when building lists.
- renamed compile-option from REGEX (to REGEXP) for consistency with
with the <regex.h>, <regex> header names etc.
wordRes
====
- renamed from wordReListMatcher -> wordRes. For reduced typing and
since it behaves as an entity only slightly related to its underlying
list nature.
- Provide old name as typedef and include for code transition.
- pass through some list methods into wordRes
hashedWordList
====
- hashedWordList[const word& name] now returns a -1 if the name is is
not found in the list of indices. That has been a pending change
ever since hashedWordList was generalized out of speciesTable
(Oct-2010).
- add operator()(const word& name) for easy use as a predicate
STYLE: adjust parameter names in stringListOps
- reflect if the parameter is being used as a primary matcher, or the
matcher will be derived from the parameter.
For example,
(const char* re), which first creates a regExp
versus (const regExp& matcher) which is used directly.
- As the names describe, check if the string starts or ends with a
particular value. Always true if the given text is empty or if the
string is identical to the given text.
- eg,
"$USER $(PWD) $USER" -> "username $(PWD) $USER"
instead of
"$USER $(PWD) $USER" -> "username $(PWD) username"
this is noticable in some dynamicCode usages
STYLE: consolidate all string expand code into stringOps
- objectRegistry gets a rename() that also adjusts the dbDir
- cloud reworked to use static variables subInstance and defaultName.
This avoids writing "lagrangian" everywhere
string fixes
- avoid masking of std::string::replace in string.H
- avoid old strstream in PV3FoamReader
- If the underlying type is contiguous, FixedList hashes its storage directly.
- Drop labelPairHash (non-commutative) from fvMeshDistribute since
FixedList::Hash does the right thing anyhow.
- Hash<edge> specialization is commutative, without multiplication.
- Hash<triFace> specialization kept multiplication (but now uLabel).
There's not much point optimizing it, since it's not used much anyhow.
Misc. changes
- added StaticAssert to NamedEnum.H
- label.H / uLabel.H : define FOAM_LABEL_MAX, FOAM_ULABEL_MAX with the
values finally used for the storage. These can be useful for pre-processor
checks elsewhere (although I stopped needing them in the meantime).
- make table power-of-two, but since it seems to give 1-2% performance
improvement, maybe forget it too.
- remove two-argument form of hashing classes and do the modulus direclty
within HashTable instead. This simplifies things a fair bit.
- migrate Hash<void*> from db/dlLibrary to primitives/hashes/Hash
- it was possible to create a PackedList::iterator from a
PackedList::const_iterator and violate const-ness
- added HashTable::printInfo for emitting some information
- changed default table sizes from 100 -> 128 in preparation for future
2^n table sizes
- string doesn't use any debug info, remove it
- restructured the macros to add in NoDebug macro versions to className,
typeInfo. Might be helpful with tackling the globals bootstrapping issue.
- HashTableName + StaticHashTableName - avoid lookup of debug switch when
FULLDEBUG is not defined
