ENH: expand UPstream communicator support

- split/duplicate functionality

- rework inter-node/intra-node handling to allow selection of
  splitting based on 'shared' or hostname (default).

- always creates node communicators at startup:
  * commInterNode() - between nodes
  * commLocalNode() - within a node

- world-comm is now always a duplicate of MPI_COMM_WORLD to provide
  better separation from other processes.

NB:
   the inter-node comm is a slight exception to other communicators
   in that we always retain its list of (global) ranks, even if the
   local process is not in that communicator.
   This can help when constructing topology-aware patterns.

FIX: non-participating ranks still had knowledge of their potential siblings

- after create by group, the procIDs_ of non-participating ranks
  should be empty (except for the inter-node exception)
This commit is contained in:
Mark Olesen
2025-02-07 13:51:39 +01:00
parent 579f8ef7c6
commit 831a55f1ba
32 changed files with 2270 additions and 1087 deletions

View File

@ -135,6 +135,18 @@ OptimisationSwitches
// Default communication type (nonBlocking | scheduled | buffered)
commsType nonBlocking;
// Use host/node topology-aware routines
// 0: disabled
// 1: split by hostname [default]
// 2: split by shared
// >=4: (debug/manual) split with given number per node
nodeComms 1;
// Minimum number of nodes before topology-aware routines are enabled
// <= 2 : always
// >= 3 : when there are more than N nodes
nodeComms.min 0;
// Transfer double as float for processor boundaries. Mostly defunct.
floatTransfer 0;