This is a map data structure rather than a class or function which performs the
mapping operation so polyMeshDistributionMap is more logical and comprehensible
than mapDistributePolyMesh.
boundaryProcAddressing has been removed. This has not been needed for a
long time. decomposePar has been optimised for mininum IO, rather than
minimum memory usage. decomposePar has also been corrected so that it
can decompose sequences of time-varying meshes.
used in conjunction with the new loadBalancing option in constant/chemistryProperties:
loadBalancing on;
which enables per-cell CPU time caching used by the loadBalancer to redistribute
the mesh. Currently this option is only provided for chemistry integration but
the implementation is general and in future options will be provided to balance
other local cell loads, in particular Lagrangian particles.
The loadBalancer in enabled by specifying a distributor entry in
constant/dynamicMeshDict, e.g.
distributor
{
type loadBalancer;
libs ("libfvMeshDistributors.so");
multiConstraint true;
// How often to redistribute
redistributionInterval 10;
// Maximum fractional cell distribution imbalance
// before rebalancing
maxImbalance 0.1;
}
with which the mesh is checked for more than 10% load-imbalance every 10
time-steps and redistributed using a multi-constraint method, i.e. separate CPU
load weights are provided for each of the loads, currently that is the chemistry
integration load and the CPU time taken for the rest of the simulation,
transport equations solution etc.
The fvMeshDistributors::loadBalancer uses the distributor specified in
system/decomposeParDict to redistribute the mesh based on the cell CPU loads,
e.g. to use the Zoltan RCB method specify:
distributor zoltan;
libs ("libzoltanDecomp.so");
zoltanCoeffs
{
lb_method rcb;
}
Unfortunately only a few available redistribution methods support
multi-constraints: Zoltan::RCB, MeTiS, parMeTiS and xtraPuLP, of these only
Zoltan::RCB is currently available in OpenFOAM. Load-balancing is possible
without using a multi-constraint method (i.e. using any of the other
decomposition methods provided with OpenFOAM and Zoltan) by summing the various
CPU loads which is selected by setting:
multiConstraint false;
but the load-balancing is likely to be a lot less effective with this option.
Due to the licencing issues with parMeTiS interfacing to xtraPuLP might be the
best option for further work on load-balancing in OpenFOAM, or MeTiS could be
used in parallel by first agglomerating the distribution graph on the master
processor and redistributing the result; this pseudo-parallel option is already
provided for the Scotch method.
Zoltan only work in parallel so zoltanDecomp can only be used for redistribution
but is much more flexible than ptscotch and provides a range of geometric, graph
and hypergraph methods which can operate in either "partition" or "repartition",
the latter being particularly useful for dynamic load-balancing by migrating
cells between processors rather than creating a completely different
decomposition, thus reducing communication.
Class
Foam::zoltanDecomp
Description
Zoltan redistribution in parallel
Note: Zoltan methods do not support serial operation.
Parameters
- lb_method : The load-balancing algorithm
- block : block partitioning
- random : random partitioning
- rcb : recursive coordinate bisection
- rib : ecursive inertial bisection
- hsfc : Hilbert space-filling curve partitioning
- reftree : refinement tree based partitioning
- graph : choose from collection of methods for graphs
- hypergraph : choose from a collection of methods for hypergraphs
- lb_approach The desired load balancing approach. Only lb_method =
hypergraph or graph uses the lb_approach parameter. Valid values are
- partition : Partition without reference to the current distribution,
recommended for static load balancing.
- repartition : Partition starting from the current data distribution
to keep data migration low, recommended for dynamic load balancing.
- refine : Quickly improve the current data distribution
Default values
- debug_level 0
- imbalance_tol 1.05
- lb_method graph
- lb_approach repartition
Usage
To select the Zoltan graph repartition method add the following entries to
decomposeParDict:
distributor zoltan;
libs ("libzoltanRenumber.so");
The Zoltan lb_method and lb_approach can be changed by adding the
corresponding entries to the optional zoltanCeoffs sub-dictionary, e.g.:
zoltanCoeffs
{
lb_method hypergraph;
lb_approach partition;
}
An example of using Zoltan for redistribution during snappyHexMesh is provided
commented out in
tutorials/incompressible/simpleFoam/motorBike/system/decomposeParDict
and fordynamic load-balancing in
tutorials/multiphase/interFoam/RAS/floatingObject/system/decomposeParDict.
Note that Zoltan must first be compiled in ThirdParty-dev by downloading from
the link in the README file and running Allwmake and then compiling zoltanDecomp
by running Allwmake in src/parallel/decompose.
When snappyHexMesh is run in parallel it re-balances the mesh during refinement
and layer addition by redistribution which requires a decomposition method
that operates in parallel, e.g. hierachical or ptscotch. decomposePar uses a
decomposition method which operates in serial e.g. hierachical but NOT
ptscotch. In order to run decomposePar followed by snappyHexMesh in parallel it
has been necessary to change the method specified in decomposeParDict but now
this is avoided by separately specifying the decomposition and distribution
methods, e.g. in the incompressible/simpleFoam/motorBike case:
numberOfSubdomains 6;
decomposer hierarchical;
distributor ptscotch;
hierarchicalCoeffs
{
n (3 2 1);
order xyz;
}
The distributor entry is also used for run-time mesh redistribution, e.g. in the
multiphase/interFoam/RAS/floatingObject case re-distribution for load-balancing
is enabled in constant/dynamicMeshDict:
distributor
{
type distributor;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
which uses the distributor specified in system/decomposeParDict:
distributor hierarchical;
This rationalisation provides the structure for development of mesh
redistribution and load-balancing.
Basic support is now provided for dynamic mesh redistribution, particularly for
load-balancing. The mesh distributor is selected in the optional 'distributor'
entry in dynamicMeshDict, for example in the
multiphase/interFoam/RAS/floatingObject tutorial case when run in parallel using
the new Allrun-parallel script
distributor
{
type decomposer;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
in which the 'decomposer' form of redistribution is selected to call the mesh
decomposition method specified in decomposeParDict to re-decompose the mesh for
redistribution. The redistributionInterval entry specifies how frequently mesh
redistribution takes place, in the above every 10th time-step. An optional
maxImbalance entry is also provided to control redistribution based on the cell
distribution imbalance:
Class
Foam::fvMeshDistributor::decomposer
Description
Dynamic mesh redistribution using the decomposer
Usage
Example of single field based refinement in all cells:
\verbatim
distributor
{
type decomposer;
libs ("libfvMeshDistributors.so");
// How often to redistribute
redistributionInterval 10;
// Maximum fractional cell distribution imbalance
// before rebalancing
maxImbalance 0.1;
}
\endverbatim
Currently mesh refinement/unrefinement and motion with redistribution is
supported but many aspects of OpenFOAM are not yet and will require further
development, in particular fvModels and Lagrangian.
Also only the geometry-based simple and hierarchical decomposition method are
well behaved for redistribution, scotch and ptScotch cause dramatic changes in
mesh distribution with a corresponding heavy communications overhead limiting
their usefulness or at least the frequency with which they should be called to
redistribute the mesh.
used to check the existence of and open an object file, read and check the
header without constructing the object.
'typeIOobject' operates in an equivalent and consistent manner to 'regIOobject'
but the type information is provided by the template argument rather than via
virtual functions for which the derived object would need to be constructed,
which is the case for 'regIOobject'.
'typeIOobject' replaces the previous separate functions 'typeHeaderOk' and
'typeFilePath' with a single consistent interface.
now all path functions in 'IOobject' are either templated on the type or require a
'globalFile' argument to specify if the type is case global e.g. 'IOdictionary' or
decomposed in parallel, e.g. almost everything else.
The 'global()' and 'globalFile()' virtual functions are now in 'regIOobject'
abstract base-class and overridden as required by derived classes. The path
functions using 'global()' and 'globalFile()' to differentiate between global
and processor local objects are now also in 'regIOobject' rather than 'IOobject'
to ensure the path returned is absolutely consistent with the type.
Unfortunately there is still potential for unexpected IO behaviour inconsistent
with the global/local nature of the type due to the 'fileOperation' classes
searching the processor directory for case global objects before searching the
case directory. This approach appears to be a work-around for incomplete
integration with and rationalisation of 'IOobject' but with the changes above it
is no longer necessary. Unfortunately this "up" searching is baked-in at a low
level and mixed-up with various complex ways to pick the processor directory
name out of the object path and will take some unravelling but this work will
undertaken as time allows.
for local and global files so that the reading and writing of local files to
processor directories and global files to the case directory are consistent.
// Reset the seed of the pseudo-random generator used by the graph
// partitioning routines of the libScotch library. Two consecutive calls to
// the same libScotch partitioning routines, and separated by a call to
// SCOTCH randomReset, will always yield the same results, as if the
// equivalent standalone Scotch programs were used twice, independently,
SCOTCH_randomReset();
This change fixes failures that occur with the mapping of fields with
patchType overrides. It fixes a crash that previously occurred when
redistributing patch fields with patchType overrides. It also makes
decomposition correctly maintain patchType overrides on cyclics when
those cyclics are separated and become processorCyclics.
These fixes have been achieved by removing the patchType override data
from the fv and point patches. Whether or not the field overrides the
underlying patchType constraint is now determined on the fly from the
patch and field names and what is available on the field run-time
selection table.
When using 'simple' or 'hierarchical' decomposition it is useful to slightly rotate a
coordinate-aligned block-mesh to improve the processor boundaries by avoiding
irregular cell distribution at those boundaries. The degree of slight rotation
is controlled by the 'delta' coefficient and a value of 0.001 is generally
suitable so to avoid unnecessary clutter in 'decomposeParDict' 'delta' now
defaults to this value.
so that it can be included directly into the wmake Makefile to allow full
support of gmake syntax, variables, functions etc.
The Make/files file handled in the same manner as the Make/options file if it
contains the SOURCE entry otherwise it is first processed by cpp for backward
compatibility.
Currently these deleted function declarations are still in the private section
of the class declarations but will be moved by hand to the public section over
time as this is too complex to automate reliably.
Replaced all uses of complex Xfer class with C++11 "move" constructors and
assignment operators. Removed the now redundant Xfer class.
This substantial changes improves consistency between OpenFOAM and the C++11 STL
containers and algorithms, reduces memory allocation and copy overhead when
returning containers from functions and simplifies maintenance of the core
libraries significantly.
Using the new field mapper framework it is now possible to create specialised
mappers rather than creating a fatter and fatter interface in the base mapper.
This approach is far more extensible, comprehensible and maintainable.
Flux fields (surfaceScalarFields) are flipped as before but this process is no
longer applied to surfaceVectorFields (Uf etc.) for which it is not appropriate.
decomposePar is now consistent with reconstructPar with respect to
surfaceVectorFields.
Patch contributed by Mattijs Janssens.
This clarifies the purpose which is to indicate that the object should be read
or written on this particular processor rather than it is or is not valid.
Setting the moving flag as a result of a reconstruction of the mesh at a
new time was causing lagrangian reconstruction to fail, as the necessary
moving mesh data had not been cached.
to rationalise the structure and class names to avoid the need for the confusing
addNamedToRunTimeSelectionTable and use instead use the standard
addToRunTimeSelectionTable to populate the run-time selection table.
