- provide a plain stream() method on messageStream to reduce reliance
on casting operators and slightly opaque operator()() calls etc
- support alternative stream for messageStream serial output.
This can be used to support local redirection of output.
For example,
refPtr<OFstream> logging; // or autoPtr, unique_ptr etc
// Later...
Info.stream(logging.get())
<< "Detailed output ..." << endl;
This will use the stdout semantics in the normal case, or allow
redirection to an output file if a target output stream is defined,
but still effectively use /dev/null on non-master processes.
This is mostly the same as this ternary
(logging ? *logging : Info())
except that the ternary could be incorrect on sub-processes,
requires more typing etc.
ENH: use case-relative names of dictionary, IOstream for FatalIOError
- normally yields more easily understandable information
- simplifies local toggling.
- centralize fileModification static variables into IOobject.
They were previously scattered between IOobject and regIOobject
- improves interface and data consistency.
Older signatures are still active (via the Foam_IOstream_extras
define).
- refine internals for IOstreamOption streamFormat, versionNumber
ENH: improve data alignment for IOstream and IOobject
- fit sizeof label/scalar into unsigned char
STYLE: remove dead code
- implicitly enabled when timeStampMaster (default) is used
for the fileModificationChecking
- When running with non-distributed roots (eg, NFS-share) read for
processor directories on master only and send to sub-processes
instead individual reads.
- If disabled (old default, or when running with distributed roots),
uses the regular fileHandler readDir, which may perform readDir
on each processor. Potentially slow startup times on large systems.
Improvements based on analysis from T.Aoyagi(RIST), A.Azami(RIST)
- previously hidden as Detail::[IO]FstreamAllocator, now exposed
directly as [io]fstreamPointer, which allows reuse for
std::ifstream, std::ofstream wrapping, without the additional
ISstream, OSstream layers.
These stream pointers have some characteristics similar to a
unique_ptr.
- restrict direct gzstream usage to two files (fstreamPointers.C,
gzstream.C) which improves localization and makes it simpler to
enable/disable with the `HAVE_LIBZ` define.
The HAVE_LIBZ define is currently simply hard-coded in the
Make/options.
If compiled WITHOUT libz support:
- reading gz files : FatalError
- writing gz files : emit warning and downgrade to uncompressed
- warn if compression is specified in the case controlDict
and downgrade to uncompressed
ENH: minor updates to gzstream interface for C++11
- support construct/open with std::string for the file names.
CONFIG: provisioning for have_libz detection as wmake/script
- clearer than passing a reference to a dummy variable,
or relying on move occuring within the copy constructor
(historical, but should be deprecated)
STYLE: consistent autoPtr syntax for uncollated file operations
* Support default values for format/compress enum lookups.
- Avoids situations where the preferred default format is not ASCII.
For example, with dictionary input:
format binar;
The typing mistake would previously have caused formatEnum to
default to ASCII. We can now properly control its behaviour.
IOstream::formatEnum
(
dict.get<word>("format"), IOstream::BINARY
);
Allowing us to switch ascii/binary, using BINARY by default even in
the case of spelling mistakes. The mistakes are flagged, but the
return value can be non-ASCII.
* The format/compression lookup behave as pass-through if the lookup
string is empty.
- Allows the following to work without complaint
IOstream::formatEnum
(
dict.getOrDefault("format", word::null), IOstream::BINARY
);
- Or use constructor-like failsafe method
IOstream::formatEnum("format", dict, IOstream::BINARY);
- Apply the same behaviour with setting stream format/compression
from a word.
is.format("binar");
will emit a warning, but leave the stream format UNCHANGED
* Rationalize versionNumber construction
- constexpr constructors where possible.
Default construct is the "currentVersion"
- Construct from token to shift the burden to versionNumber.
Support token as argument to version().
Now:
is.version(headerDict.get<token>("version"));
or failsafe constructor method
is.version
(
IOstreamOption::versionNumber("version", headerDict)
);
Before (controlled input):
is.version
(
IOstreamOption::versionNumber
(
headerDict.get<float>("version")
)
);
Old, uncontrolled input - has been removed:
is.version(headerDict.lookup("version"));
* improve consistency, default behaviour for IOstreamOption construct
- constexpr constructors where possible
- add copy construct with change of format.
- construct IOstreamOption from streamFormat is now non-explicit.
This is a commonly expected result with no ill-effects
For example,
$ someSolver -info-switch writeOptionalEntries
- note that values changed via the command-line are changed after the
etc/controlDict entries, but *before* any case-local
system/controlDict entries.
However, in many testing cases the command-line options eliminate
the need for such local file modifications.
ENH: cleanup handling of local debug switches in Time
- add as methods directly on simpleObjectRegistry to avoid code
duplication
STYLE: adjust internal naming of ITstream parameters
- as part of the cleanup of dictionary access methods (c6520033c9)
made the dictionary class single inheritance from IDLList<entry>.
This eliminates any ambiguities for iterators and allows
for simple use of range-for looping.
Eg,
for (const entry& e : topDict))
{
Info<< "entry:" << e.keyword() << " is dict:" << e.isDict() << nl;
}
vs
forAllConstIter(dictionary, topDict, iter))
{
Info<< "entry:" << iter().keyword()
<< " is dict:" << iter().isDict() << nl;
}
- more dictionary-like methods, enforce keyType::LITERAL for all
lookups to avoid any spurious keyword matching.
- new readEntry, readIfPresent methods
- The get() method replaces the now deprecate lookup() method.
- Deprecate lookupOrFailsafe()
Failsafe behaviour is now an optional parameter for lookupOrDefault,
which makes it easier to tailor behaviour at runtime.
- output of the names is now always flatted without line-breaks.
Thus,
os << flatOutput(someEnumNames.names()) << nl;
os << someEnumNames << nl;
both generate the same output.
- Constructor now uses C-string (const char*) directly instead of
Foam::word in its initializer_list.
- Remove special enum + initializer_list constructor form since
it can create unbounded lookup indices.
- Removd old hasEnum, hasName forms that were provided during initial
transition from NamedEnum.
- Added static_assert on Enum contents to restrict to enum or
integral values. Should not likely be using this class to enumerate
other things since it internally uses an 'int' for its values.
Changed volumeType accordingly to enumerate on its type (enum),
not the class itself.
- use keyType::option enum to consolidate searching options.
These enumeration names should be more intuitive to use
and improve code readability.
Eg, lookupEntry(key, keyType::REGEX);
vs lookupEntry(key, false, true);
or
Eg, lookupEntry(key, keyType::LITERAL_RECURSIVE);
vs lookupEntry(key, true, false);
- new findEntry(), findDict(), findScoped() methods with consolidated
search options for shorter naming and access names more closely
aligned with other components. Behave simliarly to the
methods lookupEntryPtr(), subDictPtr(), lookupScopedEntryPtr(),
respectively. Default search parameters consistent with lookupEntry().
Eg, const entry* e = dict.findEntry(key);
vs const entry* e = dict.lookupEntryPtr(key, false, true);
- added '*' and '->' dereference operators to dictionary searchers.
- avoids compiler ambiguity when virtual methods such as
IOdictionary::read() exist.
- the method was introduced in 1806, and was thus not yet widely used
- add additional control via a Foam::infoDetailLevel flag, which is
supported by a 'DetailLevel' macro. Eg,
DetailLevel << "some information" << nl
- When infoDetailLevel is zero, the stdout for all Foam::system() calls
are also redirected to stderr to prevent child output from
appearing on the parent.
- close stdin before exec in system call.
- controlled by the the 'printExecutionFormat' InfoSwitch in
etc/controlDict
// Style for "ExecutionTime = " output
// - 0 = seconds (with trailing 's')
// - 1 = day-hh:mm:ss
ExecutionTime = 112135.2 s ClockTime = 113017 s
ExecutionTime = 1-07:08:55.20 ClockTime = 1-07:23:37
- Callable via the new Time::printExecutionTime() method,
which also helps to reduce clutter in the applications.
Eg,
runTime.printExecutionTime(Info);
vs
Info<< "ExecutionTime = " << runTime.elapsedCpuTime() << " s"
<< " ClockTime = " << runTime.elapsedClockTime() << " s"
<< nl << endl;
--
ENH: return elapsedClockTime() and clockTimeIncrement as double
- previously returned as time_t, which is less portable.
- IOstreamOption class to encapsulate format, compression, version.
This is ordered to avoid internal padding in the structure, which
reduces several bytes of memory overhead for stream objects
and other things using this combination of data.
Byte-sizes:
old IOstream:48 PstreamBuffers:88 Time:928
new IOstream:24 PstreamBuffers:72 Time:904
====
STYLE: remove support for deprecated uncompressed/compressed selectors
In older versions, the system/controlDict used these types of
specifications:
writeCompression uncompressed;
writeCompression compressed;
As of DEC-2009, these were deprecated in favour of using normal switch
names:
writeCompression true;
writeCompression false;
writeCompression on;
writeCompression off;
Now removed these deprecated names and treat like any other unknown
input and issue a warning. Eg,
Unknown compression specifier 'compressed', assuming no compression
====
STYLE: provide Enum of stream format names (ascii, binary)
====
COMP: fixed incorrect IFstream construct in FIREMeshReader
- spurious bool argument (presumably meant as uncompressed) was being
implicitly converted to a versionNumber. Now caught by making
IOstreamOption::versionNumber constructor explicit.
- bad version specifier in changeDictionary
- add copy construct from UList
- remove copy construct from dissimilar types.
This templated constructor was too generous in what it accepted.
For the special cases where a copy constructor is required with
a change in the data type, now use the createList factory method,
which accepts a unary operator. Eg,
auto scalars = scalarList::createList
(
labels,
[](const label& val){ return 1.5*val; }
);
- expose the names of write and stopAt controls for reuse elsewhere and
provide a stopAtControls enum for 'unknown'
- track the requested number of sub-cycles (was previously a bool)
Original commit message:
------------------------
Parallel IO: New collated file format
When an OpenFOAM simulation runs in parallel, the data for decomposed fields and
mesh(es) has historically been stored in multiple files within separate
directories for each processor. Processor directories are named 'processorN',
where N is the processor number.
This commit introduces an alternative "collated" file format where the data for
each decomposed field (and mesh) is collated into a single file, which is
written and read on the master processor. The files are stored in a single
directory named 'processors'.
The new format produces significantly fewer files - one per field, instead of N
per field. For large parallel cases, this avoids the restriction on the number
of open files imposed by the operating system limits.
The file writing can be threaded allowing the simulation to continue running
while the data is being written to file. NFS (Network File System) is not
needed when using the the collated format and additionally, there is an option
to run without NFS with the original uncollated approach, known as
"masterUncollated".
The controls for the file handling are in the OptimisationSwitches of
etc/controlDict:
OptimisationSwitches
{
...
//- Parallel IO file handler
// uncollated (default), collated or masterUncollated
fileHandler uncollated;
//- collated: thread buffer size for queued file writes.
// If set to 0 or not sufficient for the file size threading is not used.
// Default: 2e9
maxThreadFileBufferSize 2e9;
//- masterUncollated: non-blocking buffer size.
// If the file exceeds this buffer size scheduled transfer is used.
// Default: 2e9
maxMasterFileBufferSize 2e9;
}
When using the collated file handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory in bytes that
is allocated. If the data exceeds this size, the write does not use threading.
When using the masterUncollated file handling, non-blocking MPI communication
requires a sufficiently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size in bytes of the buffer. If the
data exceeds this size, the system uses scheduled communication.
The installation defaults for the fileHandler choice, maxThreadFileBufferSize
and maxMasterFileBufferSize (set in etc/controlDict) can be over-ridden within
the case controlDict file, like other parameters. Additionally the fileHandler
can be set by:
- the "-fileHandler" command line argument;
- a FOAM_FILEHANDLER environment variable.
A foamFormatConvert utility allows users to convert files between the collated
and uncollated formats, e.g.
mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated
An example case demonstrating the file handling methods is provided in:
$FOAM_TUTORIALS/IO/fileHandling
The work was undertaken by Mattijs Janssens, in collaboration with Henry Weller.
to have the prefix 'write' rather than 'output'
So outputTime() -> writeTime()
but 'outputTime()' is still supported for backward-compatibility.
Also removed the redundant secondary-writing functionality from Time
which has been superseded by the 'writeRegisteredObject' functionObject.
Moved file path handling to regIOobject and made it type specific so
now every object can have its own rules. Examples:
- faceZones are now processor local (and don't search up anymore)
- timeStampMaster is now no longer hardcoded inside IOdictionary
(e.g. uniformDimensionedFields support it as well)
- the distributedTriSurfaceMesh is properly processor-local; no need
for fileModificationChecking manipulation.
