gcc-13 has new code checking and warning mechanisms which are useful but not
entirely robust and produce many false positives, particularly with respect to
local references:
warning: possibly dangling reference to a temporary
This commit resolves many of the new warning messages but the above false
warnings remain. It is possible to switch off this warning but as it also
provides some useful checks it is currently left on.
Zoltan hangs if a processor has no points/cells during redistribution and
unfortunatel Zoltan can generate distributions in which one or processors has no
cells.
The timeName() function simply returns the dimensionedScalar::name() which holds
the user-time name of the current time and now that timeName() is no longer
virtual the dimensionedScalar::name() can be called directly. The timeName()
function implementation is maintained for backward-compatibility.
This allows for partial specialisation, so the different variants of the
global IO containers do not need the function to be overloaded for each
contained type. This also fixes an ommission in providing overloads of
these functions for some of the global IO containers.
Resolves bug report https://bugs.openfoam.org/view.php?id=3890
Poly patches should not hold non-uniform physical data that needs
mapping on mesh changes (decomposition, reconstruction, topology change,
etc ...). They should only hold uniform data that can be user-specified,
or non-uniform data that can be constructed on the fly from the poly
mesh.
With the recent changes to mappedPatchBase and extrudeToRegionMesh, this
has now been consistenly enforced, and a number of incomplete
implementations of poly patch mapping have therefore been removed.
This is a very silly decomposition method that is useful for very
thoroughly testing parallelised functionality. It is absolutely not a
valid choice for any use case other than testing and debugging.
This has required implementation of finer control of stitching in the
fvMesh read constructor and readUpdate methods. Stitching is now
controlled independently of the mesh changers. Full-geometric stitching
is now always the default unless explicitly overridden in the calls to
fvMesh's read methods.
Points now always get decomposed or reconstructed for the initial face
instance. This means that they are available in the constant directory
even if decomposition/reconstruction begins at a later time which has
its own moved points field.
The reconstructPar utility now reconstructs the mesh if and when it is
necessary to do so. The reconstructParMesh utility is therefore no
longer necessary and has been removed.
It was necessary/advantagous to consolidate these utilities into one
because in the case of mesh changes it becomes increasingly less clear
which of the separate utilities is responsible for reconstructing data
that is neither clearly physical field nor mesh topology; e.g., moving
points, sets, refinement data, and so on.
avoiding problems with mesh generation, pre/post-processing applications
etc. triggering inappropriate changes to the moving and topoChanged states which
are only needed for updates in solvers corresponding to mesh changes.
The mesh will now be written only for the write time following mesh-motion or
topology change (refinement/unrefinement, mesh-to-mesh mapping, load-balancing
etc.) and not for all subsequent time-steps as it did previously. This reduces
storage and reconstruction effort of changing mesh cases.
This major development provides coupling of patches which are
non-conformal, i.e. where the faces of one patch do not match the faces
of the other. The coupling is fully conservative and second order
accurate in space, unlike the Arbitrary Mesh Interface (AMI) and
associated ACMI and Repeat AMI methods which NCC replaces.
Description:
A non-conformal couple is a connection between a pair of boundary
patches formed by projecting one patch onto the other in a way that
fills the space between them. The intersection between the projected
surface and patch forms new faces that are incorporated into the finite
volume mesh. These new faces are created identically on both sides of
the couple, and therefore become equivalent to internal faces within the
mesh. The affected cells remain closed, meaning that the area vectors
sum to zero for all the faces of each cell. Consequently, the main
benefits of the finite volume method, i.e. conservation and accuracy,
are not undermined by the coupling.
A couple connects parts of mesh that are otherwise disconnected and can
be used in the following ways:
+ to simulate rotating geometries, e.g. a propeller or stirrer, in which
a part of the mesh rotates with the geometry and connects to a
surrounding mesh which is not moving;
+ to connect meshes that are generated separately, which do not conform
at their boundaries;
+ to connect patches which only partially overlap, in which the
non-overlapped section forms another boundary, e.g. a wall;
+ to simulate a case with a geometry which is periodically repeating by
creating multiple couples with different transformations between
patches.
The capability for simulating partial overlaps replaces the ACMI
functionality, currently provided by the 'cyclicACMI' patch type, and
which is unreliable unless the couple is perfectly flat. The capability
for simulating periodically repeating geometry replaces the Repeat AMI
functionality currently provided by the 'cyclicRepeatAMI' patch type.
Usage:
The process of meshing for NCC is very similar to existing processes for
meshing for AMI. Typically, a mesh is generated with an identifiable set
of internal faces which coincide with the surface through which the mesh
will be coupled. These faces are then duplicated by running the
'createBaffles' utility to create two boundary patches. The points are
then split using 'splitBaffles' in order to permit independent motion of
the patches.
In AMI, these patches are assigned the 'cyclicAMI' patch type, which
couples them using AMI interpolation methods.
With NCC, the patches remain non-coupled, e.g. a 'wall' type. Coupling
is instead achieved by running the new 'createNonConformalCouples'
utility, which creates additional coupled patches of type
'nonConformalCyclic'. These appear in the 'constant/polyMesh/boundary'
file with zero faces; they are populated with faces in the finite volume
mesh during the connection process in NCC.
For a single couple, such as that which separates the rotating and
stationary sections of a mesh, the utility can be called using the
non-coupled patch names as arguments, e.g.
createNonConformalCouples -overwrite rotatingZoneInner rotatingZoneOuter
where 'rotatingZoneInner' and 'rotatingZoneOuter' are the names of the
patches.
For multiple couples, and/or couples with transformations,
'createNonConformalCouples' should be run without arguments. Settings
will then be read from a configuration file named
'system/createNonConformalCouplesDict'. See
'$FOAM_ETC/caseDicts/annotated/createNonConformalCouplesDict' for
examples.
Boundary conditions must be specified for the non-coupled patches. For a
couple where the patches fully overlap, boundary conditions
corresponding to a slip wall are typically applied to fields, i.e
'movingWallSlipVelocity' (or 'slip' if the mesh is stationary) for
velocity U, 'zeroGradient' or 'fixedFluxPressure' for pressure p, and
'zeroGradient' for other fields. For a couple with
partially-overlapping patches, boundary conditions are applied which
physically represent the non-overlapped region, e.g. a no-slip wall.
Boundary conditions also need to be specified for the
'nonConformalCyclic' patches created by 'createNonConformalCouples'. It
is generally recommended that this is done by including the
'$FOAM_ETC/caseDicts/setConstraintTypes' file in the 'boundaryField'
section of each of the field files, e.g.
boundaryField
{
#includeEtc "caseDicts/setConstraintTypes"
inlet
{
...
}
...
}
For moving mesh cases, it may be necessary to correct the mesh fluxes
that are changed as a result of the connection procedure. If the
connected patches do not conform perfectly to the mesh motion, then
failure to correct the fluxes can result in noise in the pressure
solution.
Correction for the mesh fluxes is enabled by the 'correctMeshPhi' switch
in the 'PIMPLE' (or equivalent) section of 'system/fvSolution'. When it
is enabled, solver settings are required for 'MeshPhi'. The solution
just needs to distribute the error enough to dissipate the noise. A
smooth solver with a loose tolerance is typically sufficient, e.g. the
settings in 'system/fvSolution' shown below:
solvers
{
MeshPhi
{
solver smoothSolver;
smoother symGaussSeidel;
tolerance 1e-2;
relTol 0;
}
...
}
PIMPLE
{
correctMeshPhi yes;
...
}
The solution of 'MeshPhi' is an inexpensive computation since it is
applied only to a small subset of the mesh adjacent to the
couple. Conservation is maintained whether or not the mesh flux
correction is enabled, and regardless of the solution tolerance for
'MeshPhi'.
Advantages of NCC:
+ NCC maintains conservation which is required for many numerical
schemes and algorithms to operate effectively, in particular those
designed to maintain boundedness of a solution.
+ Closed-volume systems no longer suffer from accumulation or loss of
mass, poor convergence of the pressure equation, and/or concentration
of error in the reference cell.
+ Partially overlapped simulations are now possible on surfaces that are
not perfectly flat. The projection fills space so no overlaps or
spaces are generated inside contiguously overlapping sections, even if
those sections have sharp angles.
+ The finite volume faces created by NCC have geometrically accurate
centres. This makes the method second order accurate in space.
+ The polyhedral mesh no longer requires duplicate boundary faces to be
generated in order to run a partially overlapped simulation.
+ Lagrangian elements can now transfer across non-conformal couplings in
parallel.
+ Once the intersection has been computed and applied to the finite
volume mesh, it can use standard cyclic or processor cyclic finite
volume boundary conditions, with no need for additional patch types or
matrix interfaces.
+ Parallel communication is done using the standard
processor-patch-field system. This is more efficient than alternative
systems since it has been carefully optimised for use within the
linear solvers.
+ Coupled patches are disconnected prior to mesh motion and topology
change and reconnected afterwards. This simplifies the boundary
condition specification for mesh motion fields.
Resolved Bug Reports:
+ https://bugs.openfoam.org/view.php?id=663
+ https://bugs.openfoam.org/view.php?id=883
+ https://bugs.openfoam.org/view.php?id=887
+ https://bugs.openfoam.org/view.php?id=1337
+ https://bugs.openfoam.org/view.php?id=1388
+ https://bugs.openfoam.org/view.php?id=1422
+ https://bugs.openfoam.org/view.php?id=1829
+ https://bugs.openfoam.org/view.php?id=1841
+ https://bugs.openfoam.org/view.php?id=2274
+ https://bugs.openfoam.org/view.php?id=2561
+ https://bugs.openfoam.org/view.php?id=3817
Deprecation:
NCC replaces the functionality provided by AMI, ACMI and Repeat AMI.
ACMI and Repeat AMI are insufficiently reliable to warrant further
maintenance so are removed in an accompanying commit to OpenFOAM-dev.
AMI is more widely used so will be retained alongside NCC for the next
version release of OpenFOAM and then subsequently removed from
OpenFOAM-dev.
This is a map data structure rather than a class or function which performs the
mapping operation so polyMeshDistributionMap is more logical and comprehensible
than mapDistributePolyMesh.
boundaryProcAddressing has been removed. This has not been needed for a
long time. decomposePar has been optimised for mininum IO, rather than
minimum memory usage. decomposePar has also been corrected so that it
can decompose sequences of time-varying meshes.
used in conjunction with the new loadBalancing option in constant/chemistryProperties:
loadBalancing on;
which enables per-cell CPU time caching used by the loadBalancer to redistribute
the mesh. Currently this option is only provided for chemistry integration but
the implementation is general and in future options will be provided to balance
other local cell loads, in particular Lagrangian particles.
The loadBalancer in enabled by specifying a distributor entry in
constant/dynamicMeshDict, e.g.
distributor
{
type loadBalancer;
libs ("libfvMeshDistributors.so");
multiConstraint true;
// How often to redistribute
redistributionInterval 10;
// Maximum fractional cell distribution imbalance
// before rebalancing
maxImbalance 0.1;
}
with which the mesh is checked for more than 10% load-imbalance every 10
time-steps and redistributed using a multi-constraint method, i.e. separate CPU
load weights are provided for each of the loads, currently that is the chemistry
integration load and the CPU time taken for the rest of the simulation,
transport equations solution etc.
The fvMeshDistributors::loadBalancer uses the distributor specified in
system/decomposeParDict to redistribute the mesh based on the cell CPU loads,
e.g. to use the Zoltan RCB method specify:
distributor zoltan;
libs ("libzoltanDecomp.so");
zoltanCoeffs
{
lb_method rcb;
}
Unfortunately only a few available redistribution methods support
multi-constraints: Zoltan::RCB, MeTiS, parMeTiS and xtraPuLP, of these only
Zoltan::RCB is currently available in OpenFOAM. Load-balancing is possible
without using a multi-constraint method (i.e. using any of the other
decomposition methods provided with OpenFOAM and Zoltan) by summing the various
CPU loads which is selected by setting:
multiConstraint false;
but the load-balancing is likely to be a lot less effective with this option.
Due to the licencing issues with parMeTiS interfacing to xtraPuLP might be the
best option for further work on load-balancing in OpenFOAM, or MeTiS could be
used in parallel by first agglomerating the distribution graph on the master
processor and redistributing the result; this pseudo-parallel option is already
provided for the Scotch method.
Zoltan only work in parallel so zoltanDecomp can only be used for redistribution
but is much more flexible than ptscotch and provides a range of geometric, graph
and hypergraph methods which can operate in either "partition" or "repartition",
the latter being particularly useful for dynamic load-balancing by migrating
cells between processors rather than creating a completely different
decomposition, thus reducing communication.
Class
Foam::zoltanDecomp
Description
Zoltan redistribution in parallel
Note: Zoltan methods do not support serial operation.
Parameters
- lb_method : The load-balancing algorithm
- block : block partitioning
- random : random partitioning
- rcb : recursive coordinate bisection
- rib : ecursive inertial bisection
- hsfc : Hilbert space-filling curve partitioning
- reftree : refinement tree based partitioning
- graph : choose from collection of methods for graphs
- hypergraph : choose from a collection of methods for hypergraphs
- lb_approach The desired load balancing approach. Only lb_method =
hypergraph or graph uses the lb_approach parameter. Valid values are
- partition : Partition without reference to the current distribution,
recommended for static load balancing.
- repartition : Partition starting from the current data distribution
to keep data migration low, recommended for dynamic load balancing.
- refine : Quickly improve the current data distribution
Default values
- debug_level 0
- imbalance_tol 1.05
- lb_method graph
- lb_approach repartition
Usage
To select the Zoltan graph repartition method add the following entries to
decomposeParDict:
distributor zoltan;
libs ("libzoltanRenumber.so");
The Zoltan lb_method and lb_approach can be changed by adding the
corresponding entries to the optional zoltanCeoffs sub-dictionary, e.g.:
zoltanCoeffs
{
lb_method hypergraph;
lb_approach partition;
}
An example of using Zoltan for redistribution during snappyHexMesh is provided
commented out in
tutorials/incompressible/simpleFoam/motorBike/system/decomposeParDict
and fordynamic load-balancing in
tutorials/multiphase/interFoam/RAS/floatingObject/system/decomposeParDict.
Note that Zoltan must first be compiled in ThirdParty-dev by downloading from
the link in the README file and running Allwmake and then compiling zoltanDecomp
by running Allwmake in src/parallel/decompose.
When snappyHexMesh is run in parallel it re-balances the mesh during refinement
and layer addition by redistribution which requires a decomposition method
that operates in parallel, e.g. hierachical or ptscotch. decomposePar uses a
decomposition method which operates in serial e.g. hierachical but NOT
ptscotch. In order to run decomposePar followed by snappyHexMesh in parallel it
has been necessary to change the method specified in decomposeParDict but now
this is avoided by separately specifying the decomposition and distribution
methods, e.g. in the incompressible/simpleFoam/motorBike case:
numberOfSubdomains 6;
decomposer hierarchical;
distributor ptscotch;
hierarchicalCoeffs
{
n (3 2 1);
order xyz;
}
The distributor entry is also used for run-time mesh redistribution, e.g. in the
multiphase/interFoam/RAS/floatingObject case re-distribution for load-balancing
is enabled in constant/dynamicMeshDict:
distributor
{
type distributor;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
which uses the distributor specified in system/decomposeParDict:
distributor hierarchical;
This rationalisation provides the structure for development of mesh
redistribution and load-balancing.
Basic support is now provided for dynamic mesh redistribution, particularly for
load-balancing. The mesh distributor is selected in the optional 'distributor'
entry in dynamicMeshDict, for example in the
multiphase/interFoam/RAS/floatingObject tutorial case when run in parallel using
the new Allrun-parallel script
distributor
{
type decomposer;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
in which the 'decomposer' form of redistribution is selected to call the mesh
decomposition method specified in decomposeParDict to re-decompose the mesh for
redistribution. The redistributionInterval entry specifies how frequently mesh
redistribution takes place, in the above every 10th time-step. An optional
maxImbalance entry is also provided to control redistribution based on the cell
distribution imbalance:
Class
Foam::fvMeshDistributor::decomposer
Description
Dynamic mesh redistribution using the decomposer
Usage
Example of single field based refinement in all cells:
\verbatim
distributor
{
type decomposer;
libs ("libfvMeshDistributors.so");
// How often to redistribute
redistributionInterval 10;
// Maximum fractional cell distribution imbalance
// before rebalancing
maxImbalance 0.1;
}
\endverbatim
Currently mesh refinement/unrefinement and motion with redistribution is
supported but many aspects of OpenFOAM are not yet and will require further
development, in particular fvModels and Lagrangian.
Also only the geometry-based simple and hierarchical decomposition method are
well behaved for redistribution, scotch and ptScotch cause dramatic changes in
mesh distribution with a corresponding heavy communications overhead limiting
their usefulness or at least the frequency with which they should be called to
redistribute the mesh.
used to check the existence of and open an object file, read and check the
header without constructing the object.
'typeIOobject' operates in an equivalent and consistent manner to 'regIOobject'
but the type information is provided by the template argument rather than via
virtual functions for which the derived object would need to be constructed,
which is the case for 'regIOobject'.
'typeIOobject' replaces the previous separate functions 'typeHeaderOk' and
'typeFilePath' with a single consistent interface.
now all path functions in 'IOobject' are either templated on the type or require a
'globalFile' argument to specify if the type is case global e.g. 'IOdictionary' or
decomposed in parallel, e.g. almost everything else.
The 'global()' and 'globalFile()' virtual functions are now in 'regIOobject'
abstract base-class and overridden as required by derived classes. The path
functions using 'global()' and 'globalFile()' to differentiate between global
and processor local objects are now also in 'regIOobject' rather than 'IOobject'
to ensure the path returned is absolutely consistent with the type.
Unfortunately there is still potential for unexpected IO behaviour inconsistent
with the global/local nature of the type due to the 'fileOperation' classes
searching the processor directory for case global objects before searching the
case directory. This approach appears to be a work-around for incomplete
integration with and rationalisation of 'IOobject' but with the changes above it
is no longer necessary. Unfortunately this "up" searching is baked-in at a low
level and mixed-up with various complex ways to pick the processor directory
name out of the object path and will take some unravelling but this work will
undertaken as time allows.
for local and global files so that the reading and writing of local files to
processor directories and global files to the case directory are consistent.
// Reset the seed of the pseudo-random generator used by the graph
// partitioning routines of the libScotch library. Two consecutive calls to
// the same libScotch partitioning routines, and separated by a call to
// SCOTCH randomReset, will always yield the same results, as if the
// equivalent standalone Scotch programs were used twice, independently,
SCOTCH_randomReset();
This change fixes failures that occur with the mapping of fields with
patchType overrides. It fixes a crash that previously occurred when
redistributing patch fields with patchType overrides. It also makes
decomposition correctly maintain patchType overrides on cyclics when
those cyclics are separated and become processorCyclics.
These fixes have been achieved by removing the patchType override data
from the fv and point patches. Whether or not the field overrides the
underlying patchType constraint is now determined on the fly from the
patch and field names and what is available on the field run-time
selection table.
When using 'simple' or 'hierarchical' decomposition it is useful to slightly rotate a
coordinate-aligned block-mesh to improve the processor boundaries by avoiding
irregular cell distribution at those boundaries. The degree of slight rotation
is controlled by the 'delta' coefficient and a value of 0.001 is generally
suitable so to avoid unnecessary clutter in 'decomposeParDict' 'delta' now
defaults to this value.
