- using the List containers, and not their low-level data_bytes(),
size_bytes() methods is more convenient and allows future
adjustments to be centralized
ENH: trivial intptr_t wrapper for MPI_Win
STYLE: minor adjustments to mpirunDebug
- UPstream::Communicator is similar to UPstream::Request to
wrap/unwrap MPI_Comm. Provides a 'lookup' method to transcribe
the internal OpenFOAM communicator tracking to the opaque wrapped
version.
- provide an 'openfoam_mpi.H' interfacing file, which includes
the <mpi.h> as well as casting routines.
Example (caution: ugly!)
MPI_Comm myComm =
PstreamUtils::Cast::to_mpi
(
UPstream::Communicator::lookup(UPstream::worldComm)
);
- now simply a no-op for out-of-range values (instead of an error),
which simplifies the calling code.
Previously
==========
if (request_ >= 0 && request_ < UPstream::nRequests())
{
UPstream::waitRequest(request_);
}
Updated
=======
UPstream::waitRequest(request_);
- when 'recycling' freed request indices, ensure they are actually
within the currently addressable range
- MPI finalization now checks outstanding requests against
MPI_REQUEST_NULL to verify that they have been waited or tested on.
Previously simply checked against freed request indices
ENH: consistent initialisation of send/receive bookkeeping
- The idea of broadcast streams is to replace multiple master to
subProcs communications with a single MPI_Bcast.
if (Pstream::master())
{
OPBstream toAll(Pstream::masterNo());
toAll << data;
}
else
{
IPBstream fromMaster(Pstream::masterNo());
fromMaster >> data;
}
// vs.
if (Pstream::master())
{
for (const int proci : Pstream::subProcs())
{
OPstream os(Pstream::commsTypes::scheduled, proci);
os << data;
}
}
else
{
IPstream is(Pstream::commsTypes::scheduled, Pstream::masterNo());
is >> data;
}
Can simply use UPstream::broadcast() directly for contiguous data
with known lengths.
Based on ideas from T.Aoyagi(RIST), A.Azami(RIST)
- native MPI min/max/sum reductions for float/double
irrespective of WM_PRECISION_OPTION
- native MPI min/max/sum reductions for (u)int32_t/(u)int64_t types,
irrespective of WM_LABEL_SIZE
- replace rarely used vector2D sum reduction with FixedList as a
indicator of its intent and also generalizes to different lengths.
OLD:
vector2D values; values.x() = ...; values.y() = ...;
reduce(values, sumOp<vector2D>());
NEW:
FixedList<scalar,2> values; values[0] = ...; values[1] = ...;
reduce(values, sumOp<scalar>());
- allow returnReduce() to use native reductions. Previous code (with
linear/tree selector) would have bypassed them inadvertently.
ENH: added support for MPI broadcast (for a memory span)
ENH: select communication schedule as a static method
- UPstream::whichCommunication(comm) to select linear/tree
communication instead of ternary or
if (Pstream::nProcs() < Pstream::nProcsSimpleSum) ...
STYLE: align nProcsSimpleSum static value with etc/controlDict override
- refactor as an MPI-independent base class.
Add bufferIPC{send,recv} private methods for construct/destruct.
Eliminates code duplication from two constructor forms and reduces
additional constructor definitions in dummy library.
- add PstreamBuffers access methods, refactor common finish sends
code, tweak member packing
ENH: resize_nocopy for processorLduInterface buffers
- content is immediately overwritten
STYLE: cull unneeded includes in processorFa*
- handled by processorLduInterface
- Provide Allwmake-mpi scripts for handling the MPI-only build segment.
- Adjust Make/options to support a FOAM_MPI_LIBBIN target location.
This will simply default to FOAM_LIBBIN/FOAM_MPI, but allows different
types of builds with out-of-tree targets.
Example,
Build OpenFOAM with default MPI settings (eg, system openmpi)
```
./Allwmake
```
Rebuild MPI-layers with different MPI.
For any given and known type
```
othermpi()
{
export WM_MPLIB=OPENMPI
export FOAM_MPI=openmpi-3.1.3
export OPAL_PREFIX=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$FOAM_MPI
export MPI_ARCH_PATH="$OPAL_PREFIX"
}
```
May wish to build/rebuild ptscotch
```
(
othermpi
$WM_THIRD_PARTY_DIR/makeSCOTCH
)
```
Rebuild the mpi-dependent parts. Can also provide install location
```
for script in $(find src -name Allwmake-mpi)
do
(
other_mpi
export FOAM_MPI_LIBBIN="$HOME/tmp/install-prefix/lib/$FOAM_MPI"
)
done
```
- drop FOAM_MPI_LIBBIN variable in favour of FOAM_MPI from which
two different locations can be determined.
Previously there was something like this:
FOAM_MPI_LIBBIN = $FOAM_LIBBIN/openmpi-1.5.1
We now instead determine the locations from the FOAM_MPI directly:
FOAM_MPI=openmpi-1.5.1
$FOAM_LIBBIN/$FOAM_MPI -> OpenFOAM mpi-related
$FOAM_EXT_LIBBIN/$FOAM_MPI -> ThirdPary mpi-related
ENH: relocate ParaView libraries and plugins back under FOAM_LIBBIN
- as discussed with Mattijs
