Zoltan only work in parallel so zoltanDecomp can only be used for redistribution
but is much more flexible than ptscotch and provides a range of geometric, graph
and hypergraph methods which can operate in either "partition" or "repartition",
the latter being particularly useful for dynamic load-balancing by migrating
cells between processors rather than creating a completely different
decomposition, thus reducing communication.
Class
Foam::zoltanDecomp
Description
Zoltan redistribution in parallel
Note: Zoltan methods do not support serial operation.
Parameters
- lb_method : The load-balancing algorithm
- block : block partitioning
- random : random partitioning
- rcb : recursive coordinate bisection
- rib : ecursive inertial bisection
- hsfc : Hilbert space-filling curve partitioning
- reftree : refinement tree based partitioning
- graph : choose from collection of methods for graphs
- hypergraph : choose from a collection of methods for hypergraphs
- lb_approach The desired load balancing approach. Only lb_method =
hypergraph or graph uses the lb_approach parameter. Valid values are
- partition : Partition without reference to the current distribution,
recommended for static load balancing.
- repartition : Partition starting from the current data distribution
to keep data migration low, recommended for dynamic load balancing.
- refine : Quickly improve the current data distribution
Default values
- debug_level 0
- imbalance_tol 1.05
- lb_method graph
- lb_approach repartition
Usage
To select the Zoltan graph repartition method add the following entries to
decomposeParDict:
distributor zoltan;
libs ("libzoltanRenumber.so");
The Zoltan lb_method and lb_approach can be changed by adding the
corresponding entries to the optional zoltanCeoffs sub-dictionary, e.g.:
zoltanCoeffs
{
lb_method hypergraph;
lb_approach partition;
}
An example of using Zoltan for redistribution during snappyHexMesh is provided
commented out in
tutorials/incompressible/simpleFoam/motorBike/system/decomposeParDict
and fordynamic load-balancing in
tutorials/multiphase/interFoam/RAS/floatingObject/system/decomposeParDict.
Note that Zoltan must first be compiled in ThirdParty-dev by downloading from
the link in the README file and running Allwmake and then compiling zoltanDecomp
by running Allwmake in src/parallel/decompose.
When snappyHexMesh is run in parallel it re-balances the mesh during refinement
and layer addition by redistribution which requires a decomposition method
that operates in parallel, e.g. hierachical or ptscotch. decomposePar uses a
decomposition method which operates in serial e.g. hierachical but NOT
ptscotch. In order to run decomposePar followed by snappyHexMesh in parallel it
has been necessary to change the method specified in decomposeParDict but now
this is avoided by separately specifying the decomposition and distribution
methods, e.g. in the incompressible/simpleFoam/motorBike case:
numberOfSubdomains 6;
decomposer hierarchical;
distributor ptscotch;
hierarchicalCoeffs
{
n (3 2 1);
order xyz;
}
The distributor entry is also used for run-time mesh redistribution, e.g. in the
multiphase/interFoam/RAS/floatingObject case re-distribution for load-balancing
is enabled in constant/dynamicMeshDict:
distributor
{
type distributor;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
which uses the distributor specified in system/decomposeParDict:
distributor hierarchical;
This rationalisation provides the structure for development of mesh
redistribution and load-balancing.
Basic support is now provided for dynamic mesh redistribution, particularly for
load-balancing. The mesh distributor is selected in the optional 'distributor'
entry in dynamicMeshDict, for example in the
multiphase/interFoam/RAS/floatingObject tutorial case when run in parallel using
the new Allrun-parallel script
distributor
{
type decomposer;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
in which the 'decomposer' form of redistribution is selected to call the mesh
decomposition method specified in decomposeParDict to re-decompose the mesh for
redistribution. The redistributionInterval entry specifies how frequently mesh
redistribution takes place, in the above every 10th time-step. An optional
maxImbalance entry is also provided to control redistribution based on the cell
distribution imbalance:
Class
Foam::fvMeshDistributor::decomposer
Description
Dynamic mesh redistribution using the decomposer
Usage
Example of single field based refinement in all cells:
\verbatim
distributor
{
type decomposer;
libs ("libfvMeshDistributors.so");
// How often to redistribute
redistributionInterval 10;
// Maximum fractional cell distribution imbalance
// before rebalancing
maxImbalance 0.1;
}
\endverbatim
Currently mesh refinement/unrefinement and motion with redistribution is
supported but many aspects of OpenFOAM are not yet and will require further
development, in particular fvModels and Lagrangian.
Also only the geometry-based simple and hierarchical decomposition method are
well behaved for redistribution, scotch and ptScotch cause dramatic changes in
mesh distribution with a corresponding heavy communications overhead limiting
their usefulness or at least the frequency with which they should be called to
redistribute the mesh.
The floatingObject tutorial has been update to demonstrate this functionality by
adding the following topoChanger entry to dynamicMeshDict:
topoChanger
{
type refiner;
libs ("libfvMeshTopoChangers.so");
// How often to refine
refineInterval 1;
// Field to be refinement on
field alpha.water;
// Refine field in between lower..upper
lowerRefineLevel 0.001;
upperRefineLevel 0.999;
// Have slower than 2:1 refinement
nBufferLayers 1;
// Refine cells only up to maxRefinement levels
maxRefinement 1;
// Stop refinement if maxCells reached
maxCells 200000;
// Flux field and corresponding velocity field. Fluxes on changed
// faces get recalculated by interpolating the velocity. Use 'none'
// on surfaceScalarFields that do not need to be reinterpolated.
correctFluxes
(
(phi none)
(nHatf none)
(rhoPhi none)
(alphaPhi.water none)
(meshPhi none)
(ghf none)
);
// Write the refinement level as a volScalarField
dumpLevel true;
}
Note that currently only single rigid body motion is supported (but multi-body
support will be added shortly) and the Crank-Nicolson scheme is not supported.
and only needed if there is a name clash between entries in the source
specification and the set specification, e.g. "name":
{
name rotorCells;
type cellSet;
action new;
source zoneToCell;
sourceInfo
{
name cylinder;
}
}
When using 'simple' or 'hierarchical' decomposition it is useful to slightly rotate a
coordinate-aligned block-mesh to improve the processor boundaries by avoiding
irregular cell distribution at those boundaries. The degree of slight rotation
is controlled by the 'delta' coefficient and a value of 0.001 is generally
suitable so to avoid unnecessary clutter in 'decomposeParDict' 'delta' now
defaults to this value.
The FOAM file format has not changed from version 2.0 in many years and so there
is no longer a need for the 'version' entry in the FoamFile header to be
required and to reduce unnecessary clutter it is now optional, defaulting to the
current file format 2.0.
A new run-time selectable interface compression scheme framework has been added
to the two-phase VoF solvers to provide greater flexibility, extensibility and
more consistent user-interface. The previously built-in interface compression
is now in the standard run-time selectable surfaceInterpolationScheme
interfaceCompression:
Class
Foam::interfaceCompression
Description
Interface compression corrected scheme, based on counter-gradient
transport, to maintain sharp interfaces during VoF simulations.
The interface compression is applied to the face interpolated field from a
suitable 2nd-order shape-preserving NVD or TVD scheme, e.g. vanLeer or
vanAlbada. A coefficient is supplied to control the degree of compression,
with a value of 1 suitable for most VoF cases to ensure interface integrity.
A value larger than 1 can be used but the additional compression can bias
the interface to follow the mesh more closely while a value smaller than 1
can lead to interface smearing.
Example:
\verbatim
divSchemes
{
.
.
div(phi,alpha) Gauss interfaceCompression vanLeer 1;
.
.
}
\endverbatim
The separate scheme for the interface compression term "div(phirb,alpha)" is no
longer required or used nor is the compression coefficient cAlpha in fvSolution
as this is now part of the "div(phi,alpha)" scheme specification as shown above.
Backward-compatibility is provided by checking the specified "div(phi,alpha)"
scheme against the known interface compression schemes and if it is not one of
those the new interfaceCompression scheme is used with the cAlpha value
specified in fvSolution.
More details can be found here:
https://cfd.direct/openfoam/free-software/multiphase-interface-capturing
Henry G. Weller
CFD Direct Ltd.
and replaced interDyMFoam with a script which reports this change.
The interDyMFoam tutorials have been moved into the interFoam directory.
This change is one of a set of developments to merge dynamic mesh functionality
into the standard solvers to improve consistency, usability, flexibility and
maintainability of these solvers.
Henry G. Weller
CFD Direct Ltd.
