Now the HashTable underlying PtrListDictionary is used for zone lookup by name
which is a lot faster than the linear search method used previously if there are
a large number of zones.
gcc-13 has new code checking and warning mechanisms which are useful but not
entirely robust and produce many false positives, particularly with respect to
local references:
warning: possibly dangling reference to a temporary
This commit resolves many of the new warning messages but the above false
warnings remain. It is possible to switch off this warning but as it also
provides some useful checks it is currently left on.
Zoltan hangs if a processor has no points/cells during redistribution and
unfortunatel Zoltan can generate distributions in which one or processors has no
cells.
This is a very silly decomposition method that is useful for very
thoroughly testing parallelised functionality. It is absolutely not a
valid choice for any use case other than testing and debugging.
This is a map data structure rather than a class or function which performs the
mapping operation so polyMeshDistributionMap is more logical and comprehensible
than mapDistributePolyMesh.
used in conjunction with the new loadBalancing option in constant/chemistryProperties:
loadBalancing on;
which enables per-cell CPU time caching used by the loadBalancer to redistribute
the mesh. Currently this option is only provided for chemistry integration but
the implementation is general and in future options will be provided to balance
other local cell loads, in particular Lagrangian particles.
The loadBalancer in enabled by specifying a distributor entry in
constant/dynamicMeshDict, e.g.
distributor
{
type loadBalancer;
libs ("libfvMeshDistributors.so");
multiConstraint true;
// How often to redistribute
redistributionInterval 10;
// Maximum fractional cell distribution imbalance
// before rebalancing
maxImbalance 0.1;
}
with which the mesh is checked for more than 10% load-imbalance every 10
time-steps and redistributed using a multi-constraint method, i.e. separate CPU
load weights are provided for each of the loads, currently that is the chemistry
integration load and the CPU time taken for the rest of the simulation,
transport equations solution etc.
The fvMeshDistributors::loadBalancer uses the distributor specified in
system/decomposeParDict to redistribute the mesh based on the cell CPU loads,
e.g. to use the Zoltan RCB method specify:
distributor zoltan;
libs ("libzoltanDecomp.so");
zoltanCoeffs
{
lb_method rcb;
}
Unfortunately only a few available redistribution methods support
multi-constraints: Zoltan::RCB, MeTiS, parMeTiS and xtraPuLP, of these only
Zoltan::RCB is currently available in OpenFOAM. Load-balancing is possible
without using a multi-constraint method (i.e. using any of the other
decomposition methods provided with OpenFOAM and Zoltan) by summing the various
CPU loads which is selected by setting:
multiConstraint false;
but the load-balancing is likely to be a lot less effective with this option.
Due to the licencing issues with parMeTiS interfacing to xtraPuLP might be the
best option for further work on load-balancing in OpenFOAM, or MeTiS could be
used in parallel by first agglomerating the distribution graph on the master
processor and redistributing the result; this pseudo-parallel option is already
provided for the Scotch method.
Zoltan only work in parallel so zoltanDecomp can only be used for redistribution
but is much more flexible than ptscotch and provides a range of geometric, graph
and hypergraph methods which can operate in either "partition" or "repartition",
the latter being particularly useful for dynamic load-balancing by migrating
cells between processors rather than creating a completely different
decomposition, thus reducing communication.
Class
Foam::zoltanDecomp
Description
Zoltan redistribution in parallel
Note: Zoltan methods do not support serial operation.
Parameters
- lb_method : The load-balancing algorithm
- block : block partitioning
- random : random partitioning
- rcb : recursive coordinate bisection
- rib : ecursive inertial bisection
- hsfc : Hilbert space-filling curve partitioning
- reftree : refinement tree based partitioning
- graph : choose from collection of methods for graphs
- hypergraph : choose from a collection of methods for hypergraphs
- lb_approach The desired load balancing approach. Only lb_method =
hypergraph or graph uses the lb_approach parameter. Valid values are
- partition : Partition without reference to the current distribution,
recommended for static load balancing.
- repartition : Partition starting from the current data distribution
to keep data migration low, recommended for dynamic load balancing.
- refine : Quickly improve the current data distribution
Default values
- debug_level 0
- imbalance_tol 1.05
- lb_method graph
- lb_approach repartition
Usage
To select the Zoltan graph repartition method add the following entries to
decomposeParDict:
distributor zoltan;
libs ("libzoltanRenumber.so");
The Zoltan lb_method and lb_approach can be changed by adding the
corresponding entries to the optional zoltanCeoffs sub-dictionary, e.g.:
zoltanCoeffs
{
lb_method hypergraph;
lb_approach partition;
}
An example of using Zoltan for redistribution during snappyHexMesh is provided
commented out in
tutorials/incompressible/simpleFoam/motorBike/system/decomposeParDict
and fordynamic load-balancing in
tutorials/multiphase/interFoam/RAS/floatingObject/system/decomposeParDict.
Note that Zoltan must first be compiled in ThirdParty-dev by downloading from
the link in the README file and running Allwmake and then compiling zoltanDecomp
by running Allwmake in src/parallel/decompose.
When snappyHexMesh is run in parallel it re-balances the mesh during refinement
and layer addition by redistribution which requires a decomposition method
that operates in parallel, e.g. hierachical or ptscotch. decomposePar uses a
decomposition method which operates in serial e.g. hierachical but NOT
ptscotch. In order to run decomposePar followed by snappyHexMesh in parallel it
has been necessary to change the method specified in decomposeParDict but now
this is avoided by separately specifying the decomposition and distribution
methods, e.g. in the incompressible/simpleFoam/motorBike case:
numberOfSubdomains 6;
decomposer hierarchical;
distributor ptscotch;
hierarchicalCoeffs
{
n (3 2 1);
order xyz;
}
The distributor entry is also used for run-time mesh redistribution, e.g. in the
multiphase/interFoam/RAS/floatingObject case re-distribution for load-balancing
is enabled in constant/dynamicMeshDict:
distributor
{
type distributor;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
which uses the distributor specified in system/decomposeParDict:
distributor hierarchical;
This rationalisation provides the structure for development of mesh
redistribution and load-balancing.
// Reset the seed of the pseudo-random generator used by the graph
// partitioning routines of the libScotch library. Two consecutive calls to
// the same libScotch partitioning routines, and separated by a call to
// SCOTCH randomReset, will always yield the same results, as if the
// equivalent standalone Scotch programs were used twice, independently,
SCOTCH_randomReset();
This change fixes failures that occur with the mapping of fields with
patchType overrides. It fixes a crash that previously occurred when
redistributing patch fields with patchType overrides. It also makes
decomposition correctly maintain patchType overrides on cyclics when
those cyclics are separated and become processorCyclics.
These fixes have been achieved by removing the patchType override data
from the fv and point patches. Whether or not the field overrides the
underlying patchType constraint is now determined on the fly from the
patch and field names and what is available on the field run-time
selection table.
When using 'simple' or 'hierarchical' decomposition it is useful to slightly rotate a
coordinate-aligned block-mesh to improve the processor boundaries by avoiding
irregular cell distribution at those boundaries. The degree of slight rotation
is controlled by the 'delta' coefficient and a value of 0.001 is generally
suitable so to avoid unnecessary clutter in 'decomposeParDict' 'delta' now
defaults to this value.
so that it can be included directly into the wmake Makefile to allow full
support of gmake syntax, variables, functions etc.
The Make/files file handled in the same manner as the Make/options file if it
contains the SOURCE entry otherwise it is first processed by cpp for backward
compatibility.
Currently these deleted function declarations are still in the private section
of the class declarations but will be moved by hand to the public section over
time as this is too complex to automate reliably.
Replaced all uses of complex Xfer class with C++11 "move" constructors and
assignment operators. Removed the now redundant Xfer class.
This substantial changes improves consistency between OpenFOAM and the C++11 STL
containers and algorithms, reduces memory allocation and copy overhead when
returning containers from functions and simplifies maintenance of the core
libraries significantly.
Using the new field mapper framework it is now possible to create specialised
mappers rather than creating a fatter and fatter interface in the base mapper.
This approach is far more extensible, comprehensible and maintainable.
Flux fields (surfaceScalarFields) are flipped as before but this process is no
longer applied to surfaceVectorFields (Uf etc.) for which it is not appropriate.
decomposePar is now consistent with reconstructPar with respect to
surfaceVectorFields.
Patch contributed by Mattijs Janssens.
