- updating the header information (by copying) was closing the stream,
removing all watches and doing a checkOut/checkIn, which could lead to
dangling references.
Now just close the stream and simply copy the IOobject header
information directly.
STYLE: mark regIOobject assignment operator as possibly deprecated
- will revisit to revise or remove in the future
- use ignore instead of seekg/tellg to swallow input (robuster)
- check for bad gcount() values
- wrap Foam::fileSize() compressed/uncompressed handling into IFstream.
- improve handling of compressed files in masterUncollatedFileOperation.
Previously read into a string via stream iterators.
Now read chunk-wise into a List of char for fewer reallocations.
- soft renames (ie, old names still available via typedefs) for more
reasonable names and more coverage with std stream variants.
The old names could be a bit cryptic.
For example, uiliststream (== an unallocated/external list storage),
which is written as std::ispanstream for C++23.
Could similarly argue that IListStream is better named as
ICharStream, since it is an input stream of characters and the
internal storage mechanism (List or something else) is mostly
irrelevant.
Extending the coverage to include all std stream variants, and
simply rewrap them for OpenFOAM IOstream types. This simplifies the
inheritance patterns and allows reuse of icharstream/ocharstream as
a drop-in replace for istringstream/ostringstream in other wrappers.
Classes:
* icharstream / ICharStream [old: none / IListStream]
* ocharstream / OCharStream [old: none / OListStream]
* ispanstream / ISpanStream [old: uiliststream / UIListStream]
* ospanstream / OSpanStream [old: none / UOListStream]
Possible new uses : read file contents into a buffer, broadcast
buffer contents to other ranks and then transfer into an icharstream
to be read from. This avoid the multiple intermediate copies that
would be associated when using an istringstream.
- Use size doubling instead of block-wise incremental for ocharstream
(OCharStream). This corresponds to the sizing behaviour as per
std::stringstream (according to gcc-11 includes)
STYLE: drop Foam_IOstream_extras constructors for memory streams
- transitional/legacy constructors but not used in any code
- can be broadly categorised as 'unthreaded'
or 'collated' (threading requirement depends on buffering)
without other opaque inheritances.
CONFIG: add hostUncollated to bash completion prompt
- use local function for the decision making, whether worldComm or a
dedicated communicator is needed (and which sibling ranks are
involved)
Co-authored-by: mattijs <mattijs>
- accept plain lists (space or comma separated) as well as the
traditional OpenFOAM lists. This simplifies argument handling
with job scripts.
For example,
simpleFoam -ioRanks 0,4,8 ...
vs
simpleFoam -ioRanks '(0 4 8)' ...
It is also possible to select the IO ranks on a per-host basis:
simpleFoam -ioRanks host ...
- expose rank/subrank handling as static fileOperation methods
- previously checked on destruction, but it is robuster to check for a
locally defined communicator during construction
- add InfoProxy output for fileOperation
ENH: add fileOperation::storeComm()
- transfers management of the communicator from external to internal.
Use with caution
- make nProcs() independent of internal storage mechanism.
- reset receive positions with finished sends
- use size of received buffers to manage validity instead of
an separate additional gather operation.
- make fileHandler deletion mechanism more
transparent by providing a nullptr signature. A nullptr parameter
is already being used in the argList destructor for shutdown, but that
relied on an implicit conversion to autoPtr to trigger things.
- improved handling of file handler replacement.
Previously had a very basic check on old vs new handlers using their
type() values (string comparison!!), which would unfortunately
prevent proper swapping of the contents.
Check the actual pointers instead.
As part of the change, treat any empty autoPtr as no-op instead of as
deletion (which is handled explicitly as nullptr instead).
In addition to making the internal logic simpler, it means that the
current file handler always changes to a valid state without
inadvertently removing everything and falling back to creating a new
default handler (again).
This handling of no-ops also simplifies call code. For example,
<code>
autoPtr<fileHandler> oldHandler;
autoPtr<fileHandler> writeHandler;
word handlerName;
if (arg.readIfPresent("writeHandler", handlerName))
{
writeHandler = fileOperation::New(handlerName);
}
oldHandler = fileHandler(std::move(writeHandler));
... do something
writeHandler = fileHandler(std::move(oldHandler));
</code>
If the "writeHandler" is not specified, each call is a no-op.
If it is specified, the handlers are swapped out each time.
- the management of the fileHandler communicators is now encapsulated
privately (managedComm_) with the final layer being responsible for
cleaning up after itself. This makes delegation/inheritance clearer
and avoids the risk of freeing an MPI communicator twice.
STYLE: uniformFile static check relocated to fileOperation layer
- recover the target of symbolic links.
This is needed when re-creating a file tree on another rank.
ENH: handle checkGzip, followLink flags in fileHander filePath()
- previously just relied on the backend defaults, now pass through
- separate init(...) for common constructor init steps
- with ATOMIC, an intermediary file is created - eg, (fileAbc~tmp~)
where all of the output is written to. When the stream goes out of
scope, this intermediary file is moved/renamed to the actually
output name - eg, (fileAbc~tmp~) -> (fileAbc).
This adds some safety if the simulation crashes while writing the
file, since it will the partial (corrupt) file will be left
behind as (fileAbc~tmp~) and not as (fileAbc), which means it will
will be treated as a backup file and not loaded again on restart.
ENH: provided enumeration for APPEND/NON_APPEND
- clearer than using bool (with comments).
Since append mode is primarily only used by masterOFstream etc
this change is unlikely to affect user coding.
ENH: use file atomic for ensight file creation
- avoids corrupt (truncated) files being referenced by the ensight
case file if the simulation crashes while writing the ensight file.
ENH: extend rmDir to handle removal of empty directories only
- recursively remove directories that only contain other directories
but no other contents. Treats dead links as non-content.
- use DynamicList instead of List in the cache, which reduces the
number of allocations occuring each time.
- since the cached times are stored in sorted order, first check if the
new time is greater than the last list entry. Can then simply append
without performing a binary search and can obviously also skip any
subsequent sorting.
STYLE: add noexcept to Instant methods, declare in header (like Tuple2)
- adds handling of negative start times for masterUncollatedFileOperation
as well (#1112).
- handle failures *after* restoring non-parRun mode.
This ensures exit(FatalError) will exit MPI properly as well.
STYLE: replace "polyMesh" with polyMesh::meshSubDir
STYLE: adjust IOobject read/write enumerated values
- provision for possible bitwise handling
- additional Pstream::broadcasts() method to serialize/deserialize
multiple items.
- revoke the broadcast specialisations for std::string and List(s) and
use a generic broadcasting template. In most cases, the previous
specialisations would have required two broadcasts:
(1) for the size
(2) for the contiguous content.
Now favour reduced communication over potential local (intermediate)
storage that would have only benefited a few select cases.
ENH: refine PstreamBuffers access methods
- replace 'bool hasRecvData(label)' with 'label recvDataCount(label)'
to recover the number of unconsumed receive bytes from specified
processor. Can use 'labelList recvDataCounts()' to recover the
number of unconsumed receive bytes from all processor.
- additional peekRecvData() method (for transcribing contiguous data)
ENH: globalIndex whichProcID - check for isLocal first
- reasonable to assume that local items are searched for more
frequently, so do preliminary check for isLocal before performing
a more costly binary search of globalIndex offsets
ENH: masterUncollatedFileOperation - bundled scatter of status
BUG: masterUncollatedFileOperation checking of file-size
- used Foam:fileSize check to decide on scheduled/nonBlocking but this
was being done on all ranks and subsequently broadcast.
Now avoid unnecessary filesystem access on non-master ranks.
- split off a Pstream::genericBroadcast() which uses UOPBstream during
serialization and UOPBstream during de-serialization.
This function will not normally be used directly by callers, but
provides a base layer for higher-level broadcast calls.
- low-level UPstream broadcast of string content.
Since std::string has length and contiguous content, it is possible
to handle directly by the following:
1. broadcast size
2. resize
3. broadcast content when size != 0
Although this is a similar amount of communication as the generic
streaming version (min 1, max 2 broadcasts) it is more efficient
by avoiding serialization/de-serialization overhead.
- handle broadcast of List content distinctly.
Allows an optimized path for contiguous data, similar to how
std::string is handled (broadcast size, resize container, broadcast
content when size != 0), but can revert to genericBroadcast (streamed)
for non-contiguous data.
- make various scatter variants simple aliases for broadcast, since
that is what they are doing behind the scenes anyhow:
* scatter()
* combineScatter()
* listCombineScatter()
* mapCombineScatter()
Except scatterList() which remains somewhat different.
Beyond the additional (size == nProcs) check, the only difference to
using broadcast(List<T>&) or a regular scatter(List<T>&) is that
processor-local data is skipped. So leave this variant as-is.
STYLE: rename/prefix implementation code with 'Pstream'
- better association with its purpose and provides a unique name
- native MPI min/max/sum reductions for float/double
irrespective of WM_PRECISION_OPTION
- native MPI min/max/sum reductions for (u)int32_t/(u)int64_t types,
irrespective of WM_LABEL_SIZE
- replace rarely used vector2D sum reduction with FixedList as a
indicator of its intent and also generalizes to different lengths.
OLD:
vector2D values; values.x() = ...; values.y() = ...;
reduce(values, sumOp<vector2D>());
NEW:
FixedList<scalar,2> values; values[0] = ...; values[1] = ...;
reduce(values, sumOp<scalar>());
- allow returnReduce() to use native reductions. Previous code (with
linear/tree selector) would have bypassed them inadvertently.
ENH: added support for MPI broadcast (for a memory span)
ENH: select communication schedule as a static method
- UPstream::whichCommunication(comm) to select linear/tree
communication instead of ternary or
if (Pstream::nProcs() < Pstream::nProcsSimpleSum) ...
STYLE: align nProcsSimpleSum static value with etc/controlDict override
- simplifies local toggling.
- centralize fileModification static variables into IOobject.
They were previously scattered between IOobject and regIOobject
- currently add to mesh zones to provide a table of contents
of the zone names that allows downstream consumers quick access to
the information without needing to parse the entire file.
- support selective enable/disable of the file banner.
ENH: improve code isolation for decomposedBlockData
- use readBlockEntry/writeBlockEntry to encapsulate the IO handling,
which ensures more consistency
- new decomposedBlockData::readHeader for chaining into the
block header information.
- remove unused constructors for decomposedBlockData
ENH: minor cleanup of collated fileOperations
- improves interface and data consistency.
Older signatures are still active (via the Foam_IOstream_extras
define).
- refine internals for IOstreamOption streamFormat, versionNumber
ENH: improve data alignment for IOstream and IOobject
- fit sizeof label/scalar into unsigned char
STYLE: remove dead code
- robuster matching behaviour when encountering paths that themselves
contain the word "processor" in them. For example,
"/path/processor0generation2/case1/processor10/system"
will now correctly match on processor10 instead of failing.
- use procRangeType for encapsulating the processor ranges
- provision for information of distributed vs non-distributed roots.
The information is currently available from the initial setup, but
can useful to access directly within fileOperation.
STYLE: modernize list iteration
- returns a range of `int` values that can be iterated across.
For example,
for (const int proci : Pstream::subProcs()) { ... }
instead of
for
(
int proci = Pstream::firstSlave();
proci <= Pstream::lastSlave();
++proci
)
{
...
}
- returns a range of `int` values that can be iterated across.
For example,
for (const int proci : Pstream::allProcs()) { ... }
instead of
for (label proci = 0; proci < Pstream::nProcs(); ++proci) { ... }
- improves flexibility. Can tag a tmp as allowing non-const access to
the reference and skip additional const_cast in following code. For
example,
tmp<volScalarField> tfld(nullptr);
auto* ptr = getObjectPtr<volScalarField>("field");
if (ptr)
{
tfld.ref(*ptr);
}
else
{
tfld.reset(volScalarField::New(...));
}
auto& fld = tfld.ref();
ENH: renamed tmpNrc to refPtr
- the name 'refPtr' (reference|pointer) should be easier to remember
than tmpNrc (tmp, but non-ref-counted).
- provide tmpNrc typedef and header for code compatibility
NOTE
- in some places refPtr and tmp can be used instead of a
std::reference_wrapper for handling external references.
Unlike std::reference_wrapper, it can be default constructed
(holding nothing), whereas reference_wrapper may need a dummy
reference. However, the lifetime extension of references _may_ be
better with reference_wrapper.
- clearer than passing a reference to a dummy variable,
or relying on move occuring within the copy constructor
(historical, but should be deprecated)
STYLE: consistent autoPtr syntax for uncollated file operations
