- previously there were two places that used nProcsSimpleSum for
defining the communication pattern. However, this meant in practice
that filling of linearCommunication_ and treeCommunication_ themselves
where globally controlled.
Now retain the state within commsStructList and use linear/tree
when populating.
- deprecated and replaced (JAN-2023) with a version that uses
UPstream::Request to track the request. This avoids placing a
request on the global list and then having difficulty isolating it
later (or having it cleared out by someone else).
ENH: UPstream::addRequest() now ignores null requests
- avoids placing useless requests on the global list
- replace with plusOp for reductions. The old use didn't necessarily work
well with compiler resolution in all cases.
Simplify to use plusOp (and removed the old version).
[Does not affect very much code...]
COMP: incorrect include ordering for GeometricFieldFunctions
- header was included after the template code
REGRESSION: combineAllGather mapped to incorrect method (81fa7d08ee)
STYLE: use UPstream::commWarn(...) setter method
- eliminates nearly identical code between 'gather' and 'combineGather'
* Normal gather updates by assigning the result of the binary operation.
* Combine gather updates by using a binary operator that modifies
its first parameter in-place
By-product of this refactoring are these new variants:
listGather(), listGatherReduce()
mapGather(), mapGatherReduce()
that mirror the previously existing ones
listCombineGather(), listCombineReduce()
mapCombineGather(), mapCombineReduce()
except that they use the 'regular' binary operator
- the general OpenFOAM way of collecting data often looks like this:
List<scalar> allValues(numProcs);
allValues[myProc] = localValue;
Pstream::gatherList(allValues);
// Now possible like this
List<scalar> allValues(numProcs);
allValues[myProc] = localValue;
UPstream::mpiGather(nullptr, allValues.data_bytes(), sizeof(scalar));
- adjusted Pstream::gatherList accordingly.
Split out the manual implementations, give them new names
(..._tree_algorithm) and make them private.
STYLE: rename UPstream::gather -> UPstream::mpiGatherv
- easier to identify and establishes the connection to the MPI call
- additional startup guard for inter-node/local-node queries (UPstream)
- impose linear communication tree for inter-node/local-node
communicators. Was previously defaulted to a basic tree, but more
consistent to have flat addressing for these types of connections.
- demand-driven UPstream::interNode_offsets() for walking
inter-node ranges instead of creating it manually in various places.
- (style): List<int> instead of labelList for internal commsStruct
since the communication structures are tied to MPI sizes
and not to the OpenFOAM label sizes
- reduce the number of intermediate buffer allocations within
gatherList, scatterList.
- replace gatherValues() with listGatherValues(), which returns the
gathered values instead of passing by argument. This makes it less
fragile and more convenient. Naming as per
Pstream::listGatherValues(), but still retaining the original
parameter order.
- rename inplace 'gather' variant as 'gatherInplace' for clarity.
- changed signature of lowest-level globalIndex::gather routines from
List<Type> to UList<Type> for the output and removed any resizing.
This means that the caller is responsible for ensuring that the
receiving field is adequately sized (on master). This change allows
potential reuse of 'work' arrays etc.
The only code change for using this low-level gather was in
GAMGAgglomeration.
- split/duplicate functionality
- rework inter-node/intra-node handling to allow selection of
splitting based on 'shared' or hostname (default).
- always creates node communicators at startup:
* commInterNode() - between nodes
* commLocalNode() - within a node
- world-comm is now always a duplicate of MPI_COMM_WORLD to provide
better separation from other processes.
NB:
the inter-node comm is a slight exception to other communicators
in that we always retain its list of (global) ranks, even if the
local process is not in that communicator.
This can help when constructing topology-aware patterns.
FIX: non-participating ranks still had knowledge of their potential siblings
- after create by group, the procIDs_ of non-participating ranks
should be empty (except for the inter-node exception)
- using the List containers, and not their low-level data_bytes(),
size_bytes() methods is more convenient and allows future
adjustments to be centralized
ENH: trivial intptr_t wrapper for MPI_Win
STYLE: minor adjustments to mpirunDebug
- 'if constexpr (...)'
* instead of std::enable_if
* terminate template recursion
* compile-time elimination of code
- use C++14 '_t', '_v' versions,
eg, std::is_integral_v<T> instead of std::is_integral<T>::value
- std::begin, std::end, std::void_t instead of prev stdFoam versions
- provide is_contiguous_v<..> as short form of is_contiguous<..>::value
with the additional benefit of removing any cv qualifiers.
ENH: include is_rotational_vectorspace trait
- tests for vector-space and nComponents > 1 (ie, not sphericalTensor)
ENH: improve robustness of pTraits_.. tests by removing cv qualifiers
- as per std::array and FixedList
STYLE: rename Scalar.[CH] -> scalarImpl.[CH] (these are internal files)
- added inclusion guards to protect against bad use
- poor naming (eg, 'exponential.[CH]') within the same set of library
files means that the lnInclude/ contents are inconsistent or partly
meaningless (related to #2520 and !542).
Remedied by adding model-type qualifiers to the file names.
- remove unneeded -Wno-old-style flags with boost/cgal elements
- add nvc equivalent to gcc/clang -Wno-invalid-offsetof
- avoid warnings about set-but-unused variable and change of scoping
STYLE: remove wmake 'CGAL' rule
- was a transitional forward to 'cgal' rule since DEC-2020
Description:
The plicRDF interface reconstruction scheme calculates a reconstructed distance
function (RDF) in all interface cells and their point neighbours.
In point neighbours to interface cells, the RDF is calculated as a weighted
average of the distances to all adjacent interface cell centres with the weight
being the inverse distance to the cell centre.
By using the zoneDistribute class written by Henning Scheufler, the required
stencil data is communicated efficiently for stencil cells living on different
sides of one or more processor patches.
Some of the data required for the RDF reconstruction are cell centre and
interface centre positions. When a stencil extends across a cyclic patch these
positions have so far not been properly transformed in OpenFOAM. This issue is
fixed by the current contribution.
The fix is done by modifying the zoneDistribute class to hold the required
information about zone cells adjacent to cyclic patches. Positions are then
communicated with a new getPosition function which replaces getValue for
position data in the reconstructedDistanceFunction and plicRDF classes.
The implementation does not change the behaviour for cells not on
a cyclic patch and should have insignificant effect on efficiency for these.
The implementation can probably be optimised in terms of efficiency for zone
cells on cyclic patches, but we note that there are generally only very few of
these (interface cells and their point neighbours on cyclic patches) and so
the potential for speedup is expected to be limited.
Current limitations:
- In parallel, the user must constrain the decomposition to preserve cyclic
patches on the same processor, for the implementation to work properly.
- See an example here: tutorials/discInConstantFlowCyclicBCs/system/decomposeParDict
- In the case of parallel rotational cyclics that are split by the decomposition
the current bugfix does not work and therefore throws an error. This is ongoing
work and should be reported and fixed by a future patch.
For further details, please see the modified files and the comments therein:
- $FOAM_SRC/finiteVolume/fvMesh/zoneDistribute/zoneDistribute.H
- $FOAM_SRC/finiteVolume/fvMesh/zoneDistribute/zoneDistribute.C
- $FOAM_SRC/finiteVolume/fvMesh/zoneDistribute/zoneDistributeI.H
- $FOAM_SRC/transportModels/geometricVoF/reconstructedDistanceFunction/reconstructedDistanceFunction.C
- $FOAM_SRC/src/transportModels/geometricVoF/reconstructionSchemes/plicSchemes/plicRDF/plicRDF.C
Co-authored-by: David Müller <> KIT
Co-authored-by: Konstantinos Missios <> Roskilde University
Co-authored-by: Johan Roenby <> Roskilde University and STROMNING
