This major development provides coupling of patches which are
non-conformal, i.e. where the faces of one patch do not match the faces
of the other. The coupling is fully conservative and second order
accurate in space, unlike the Arbitrary Mesh Interface (AMI) and
associated ACMI and Repeat AMI methods which NCC replaces.
Description:
A non-conformal couple is a connection between a pair of boundary
patches formed by projecting one patch onto the other in a way that
fills the space between them. The intersection between the projected
surface and patch forms new faces that are incorporated into the finite
volume mesh. These new faces are created identically on both sides of
the couple, and therefore become equivalent to internal faces within the
mesh. The affected cells remain closed, meaning that the area vectors
sum to zero for all the faces of each cell. Consequently, the main
benefits of the finite volume method, i.e. conservation and accuracy,
are not undermined by the coupling.
A couple connects parts of mesh that are otherwise disconnected and can
be used in the following ways:
+ to simulate rotating geometries, e.g. a propeller or stirrer, in which
a part of the mesh rotates with the geometry and connects to a
surrounding mesh which is not moving;
+ to connect meshes that are generated separately, which do not conform
at their boundaries;
+ to connect patches which only partially overlap, in which the
non-overlapped section forms another boundary, e.g. a wall;
+ to simulate a case with a geometry which is periodically repeating by
creating multiple couples with different transformations between
patches.
The capability for simulating partial overlaps replaces the ACMI
functionality, currently provided by the 'cyclicACMI' patch type, and
which is unreliable unless the couple is perfectly flat. The capability
for simulating periodically repeating geometry replaces the Repeat AMI
functionality currently provided by the 'cyclicRepeatAMI' patch type.
Usage:
The process of meshing for NCC is very similar to existing processes for
meshing for AMI. Typically, a mesh is generated with an identifiable set
of internal faces which coincide with the surface through which the mesh
will be coupled. These faces are then duplicated by running the
'createBaffles' utility to create two boundary patches. The points are
then split using 'splitBaffles' in order to permit independent motion of
the patches.
In AMI, these patches are assigned the 'cyclicAMI' patch type, which
couples them using AMI interpolation methods.
With NCC, the patches remain non-coupled, e.g. a 'wall' type. Coupling
is instead achieved by running the new 'createNonConformalCouples'
utility, which creates additional coupled patches of type
'nonConformalCyclic'. These appear in the 'constant/polyMesh/boundary'
file with zero faces; they are populated with faces in the finite volume
mesh during the connection process in NCC.
For a single couple, such as that which separates the rotating and
stationary sections of a mesh, the utility can be called using the
non-coupled patch names as arguments, e.g.
createNonConformalCouples -overwrite rotatingZoneInner rotatingZoneOuter
where 'rotatingZoneInner' and 'rotatingZoneOuter' are the names of the
patches.
For multiple couples, and/or couples with transformations,
'createNonConformalCouples' should be run without arguments. Settings
will then be read from a configuration file named
'system/createNonConformalCouplesDict'. See
'$FOAM_ETC/caseDicts/annotated/createNonConformalCouplesDict' for
examples.
Boundary conditions must be specified for the non-coupled patches. For a
couple where the patches fully overlap, boundary conditions
corresponding to a slip wall are typically applied to fields, i.e
'movingWallSlipVelocity' (or 'slip' if the mesh is stationary) for
velocity U, 'zeroGradient' or 'fixedFluxPressure' for pressure p, and
'zeroGradient' for other fields. For a couple with
partially-overlapping patches, boundary conditions are applied which
physically represent the non-overlapped region, e.g. a no-slip wall.
Boundary conditions also need to be specified for the
'nonConformalCyclic' patches created by 'createNonConformalCouples'. It
is generally recommended that this is done by including the
'$FOAM_ETC/caseDicts/setConstraintTypes' file in the 'boundaryField'
section of each of the field files, e.g.
boundaryField
{
#includeEtc "caseDicts/setConstraintTypes"
inlet
{
...
}
...
}
For moving mesh cases, it may be necessary to correct the mesh fluxes
that are changed as a result of the connection procedure. If the
connected patches do not conform perfectly to the mesh motion, then
failure to correct the fluxes can result in noise in the pressure
solution.
Correction for the mesh fluxes is enabled by the 'correctMeshPhi' switch
in the 'PIMPLE' (or equivalent) section of 'system/fvSolution'. When it
is enabled, solver settings are required for 'MeshPhi'. The solution
just needs to distribute the error enough to dissipate the noise. A
smooth solver with a loose tolerance is typically sufficient, e.g. the
settings in 'system/fvSolution' shown below:
solvers
{
MeshPhi
{
solver smoothSolver;
smoother symGaussSeidel;
tolerance 1e-2;
relTol 0;
}
...
}
PIMPLE
{
correctMeshPhi yes;
...
}
The solution of 'MeshPhi' is an inexpensive computation since it is
applied only to a small subset of the mesh adjacent to the
couple. Conservation is maintained whether or not the mesh flux
correction is enabled, and regardless of the solution tolerance for
'MeshPhi'.
Advantages of NCC:
+ NCC maintains conservation which is required for many numerical
schemes and algorithms to operate effectively, in particular those
designed to maintain boundedness of a solution.
+ Closed-volume systems no longer suffer from accumulation or loss of
mass, poor convergence of the pressure equation, and/or concentration
of error in the reference cell.
+ Partially overlapped simulations are now possible on surfaces that are
not perfectly flat. The projection fills space so no overlaps or
spaces are generated inside contiguously overlapping sections, even if
those sections have sharp angles.
+ The finite volume faces created by NCC have geometrically accurate
centres. This makes the method second order accurate in space.
+ The polyhedral mesh no longer requires duplicate boundary faces to be
generated in order to run a partially overlapped simulation.
+ Lagrangian elements can now transfer across non-conformal couplings in
parallel.
+ Once the intersection has been computed and applied to the finite
volume mesh, it can use standard cyclic or processor cyclic finite
volume boundary conditions, with no need for additional patch types or
matrix interfaces.
+ Parallel communication is done using the standard
processor-patch-field system. This is more efficient than alternative
systems since it has been carefully optimised for use within the
linear solvers.
+ Coupled patches are disconnected prior to mesh motion and topology
change and reconnected afterwards. This simplifies the boundary
condition specification for mesh motion fields.
Resolved Bug Reports:
+ https://bugs.openfoam.org/view.php?id=663
+ https://bugs.openfoam.org/view.php?id=883
+ https://bugs.openfoam.org/view.php?id=887
+ https://bugs.openfoam.org/view.php?id=1337
+ https://bugs.openfoam.org/view.php?id=1388
+ https://bugs.openfoam.org/view.php?id=1422
+ https://bugs.openfoam.org/view.php?id=1829
+ https://bugs.openfoam.org/view.php?id=1841
+ https://bugs.openfoam.org/view.php?id=2274
+ https://bugs.openfoam.org/view.php?id=2561
+ https://bugs.openfoam.org/view.php?id=3817
Deprecation:
NCC replaces the functionality provided by AMI, ACMI and Repeat AMI.
ACMI and Repeat AMI are insufficiently reliable to warrant further
maintenance so are removed in an accompanying commit to OpenFOAM-dev.
AMI is more widely used so will be retained alongside NCC for the next
version release of OpenFOAM and then subsequently removed from
OpenFOAM-dev.
boundaryProcAddressing has been removed. This has not been needed for a
long time. decomposePar has been optimised for mininum IO, rather than
minimum memory usage. decomposePar has also been corrected so that it
can decompose sequences of time-varying meshes.
replacing the virtual functions overridden in engineTime.
Now the userTime conversion function in Time is specified in system/controlDict
such that the solver as well as all pre- and post-processing tools also operate
correctly with the chosen user-time.
For example the user-time and rpm in the tutorials/combustion/XiEngineFoam/kivaTest case are
now specified in system/controlDict:
userTime
{
type engine;
rpm 1500;
}
The default specification is real-time:
userTime
{
type real;
}
but this entry can be omitted as the real-time class is instantiated
automatically if the userTime entry is not present in system/controlDict.
Mesh motion and topology change are now combinable run-time selectable options
within fvMesh, replacing the restrictive dynamicFvMesh which supported only
motion OR topology change.
All solvers which instantiated a dynamicFvMesh now instantiate an fvMesh which
reads the optional constant/dynamicFvMeshDict to construct an fvMeshMover and/or
an fvMeshTopoChanger. These two are specified within the optional mover and
topoChanger sub-dictionaries of dynamicFvMeshDict.
When the fvMesh is updated the fvMeshTopoChanger is first executed which can
change the mesh topology in anyway, adding or removing points as required, for
example for automatic mesh refinement/unrefinement, and all registered fields
are mapped onto the updated mesh. The fvMeshMover is then executed which moved
the points only and calculates the cell volume change and corresponding
mesh-fluxes for conservative moving mesh transport. If multiple topological
changes or movements are required these would be combined into special
fvMeshMovers and fvMeshTopoChangers which handle the processing of a list of
changes, e.g. solidBodyMotionFunctions:multiMotion.
The tutorials/multiphase/interFoam/laminar/sloshingTank3D3DoF case has been
updated to demonstrate this new functionality by combining solid-body motion
with mesh refinement/unrefinement:
/*--------------------------------*- C++ -*----------------------------------*\
========= |
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\\ / O peration | Website: https://openfoam.org
\\ / A nd | Version: dev
\\/ M anipulation |
\*---------------------------------------------------------------------------*/
FoamFile
{
format ascii;
class dictionary;
location "constant";
object dynamicMeshDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
mover
{
type motionSolver;
libs ("libfvMeshMovers.so" "libfvMotionSolvers.so");
motionSolver solidBody;
solidBodyMotionFunction SDA;
CofG (0 0 0);
lamda 50;
rollAmax 0.2;
rollAmin 0.1;
heaveA 4;
swayA 2.4;
Q 2;
Tp 14;
Tpn 12;
dTi 0.06;
dTp -0.001;
}
topoChanger
{
type refiner;
libs ("libfvMeshTopoChangers.so");
// How often to refine
refineInterval 1;
// Field to be refinement on
field alpha.water;
// Refine field in between lower..upper
lowerRefineLevel 0.001;
upperRefineLevel 0.999;
// Have slower than 2:1 refinement
nBufferLayers 1;
// Refine cells only up to maxRefinement levels
maxRefinement 1;
// Stop refinement if maxCells reached
maxCells 200000;
// Flux field and corresponding velocity field. Fluxes on changed
// faces get recalculated by interpolating the velocity. Use 'none'
// on surfaceScalarFields that do not need to be reinterpolated.
correctFluxes
(
(phi none)
(nHatf none)
(rhoPhi none)
(alphaPhi.water none)
(meshPhi none)
(meshPhi_0 none)
(ghf none)
);
// Write the refinement level as a volScalarField
dumpLevel true;
}
// ************************************************************************* //
Note that currently this is the only working combination of mesh-motion with
topology change within the new framework and further development is required to
update the set of topology changers so that topology changes with mapping are
separated from the mesh-motion so that they can be combined with any of the
other movements or topology changes in any manner.
All of the solvers and tutorials have been updated to use the new form of
dynamicMeshDict but backward-compatibility was not practical due to the complete
reorganisation of the mesh change structure.
for
db/functionObjects/timeControl/timeControl.H: timeControls
primitives/bools/Switch/Switch.H: class switchType
primitives/strings/fileName/fileName.H: fileType
primitives/strings/wordRe/wordRe.H: compOption
Improvements to existing functionality
--------------------------------------
- MPI is initialised without thread support if it is not needed e.g. uncollated
- Use native c++11 threading; avoids problem with static destruction order.
- etc/cellModels now only read if needed.
- etc/controlDict can now be read from the environment variable FOAM_CONTROLDICT
- Uniform files (e.g. '0/uniform/time') are now read only once on the master only
(with the masterUncollated or collated file handlers)
- collated format writes to 'processorsNNN' instead of 'processors'. The file
format is unchanged.
- Thread buffer and file buffer size are no longer limited to 2Gb.
The global controlDict file contains parameters for file handling. Under some
circumstances, e.g. running in parallel on a system without NFS, the user may
need to set some parameters, e.g. fileHandler, before the global controlDict
file is read from file. To support this, OpenFOAM now allows the global
controlDict to be read as a string set to the FOAM_CONTROLDICT environment
variable.
The FOAM_CONTROLDICT environment variable can be set to the content the global
controlDict file, e.g. from a sh/bash shell:
export FOAM_CONTROLDICT=$(foamDictionary $FOAM_ETC/controlDict)
FOAM_CONTROLDICT can then be passed to mpirun using the -x option, e.g.:
mpirun -np 2 -x FOAM_CONTROLDICT simpleFoam -parallel
Note that while this avoids the need for NFS to read the OpenFOAM configuration
the executable still needs to load shared libraries which must either be copied
locally or available via NFS or equivalent.
New: Multiple IO ranks
----------------------
The masterUncollated and collated fileHandlers can now use multiple ranks for
writing e.g.:
mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)'
In this example ranks 0 ('processor0') and 3 ('processor3') now handle all the
I/O. Rank 0 handles 0,1,2 and rank 3 handles 3,4,5. The set of IO ranks should always
include 0 as first element and be sorted in increasing order.
The collated fileHandler uses the directory naming processorsNNN_XXX-YYY where
NNN is the total number of processors and XXX and YYY are first and last
processor in the rank, e.g. in above example the directories would be
processors6_0-2
processors6_3-5
and each of the collated files in these contains data of the local ranks
only. The same naming also applies when e.g. running decomposePar:
decomposePar -fileHandler collated -ioRanks '(0 3)'
New: Distributed data
---------------------
The individual root directories can be placed on different hosts with different
paths if necessary. In the current framework it is necessary to specify the
root per slave process but this has been simplified with the option of specifying
the root per host with the -hostRoots command line option:
mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)' \
-hostRoots '("machineA" "/tmp/" "machineB" "/tmp")'
The hostRoots option is followed by a list of machine name + root directory, the
machine name can contain regular expressions.
New: hostCollated
-----------------
The new hostCollated fileHandler automatically sets the 'ioRanks' according to
the host name with the lowest rank e.g. to run simpleFoam on 6 processors with
ranks 0-2 on machineA and ranks 3-5 on machineB with the machines specified in
the hostfile:
mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler hostCollated
This is equivalent to
mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler collated -ioRanks '(0 3)'
This example will write directories:
processors6_0-2/
processors6_3-5/
A typical example would use distributed data e.g. no two nodes, machineA and
machineB, each with three processes:
decomposePar -fileHandler collated -case cavity
# Copy case (constant/*, system/*, processors6/) to master:
rsync -a cavity machineA:/tmp/
# Create root on slave:
ssh machineB mkdir -p /tmp/cavity
# Run
mpirun --hostfile hostfile icoFoam \
-case /tmp/cavity -parallel -fileHandler hostCollated \
-hostRoots '("machineA" "/tmp" "machineB" "/tmp")'
Contributed by Mattijs Janssens
When an OpenFOAM simulation runs in parallel, the data for decomposed fields and
mesh(es) has historically been stored in multiple files within separate
directories for each processor. Processor directories are named 'processorN',
where N is the processor number.
This commit introduces an alternative "collated" file format where the data for
each decomposed field (and mesh) is collated into a single file, which is
written and read on the master processor. The files are stored in a single
directory named 'processors'.
The new format produces significantly fewer files - one per field, instead of N
per field. For large parallel cases, this avoids the restriction on the number
of open files imposed by the operating system limits.
The file writing can be threaded allowing the simulation to continue running
while the data is being written to file. NFS (Network File System) is not
needed when using the the collated format and additionally, there is an option
to run without NFS with the original uncollated approach, known as
"masterUncollated".
The controls for the file handling are in the OptimisationSwitches of
etc/controlDict:
OptimisationSwitches
{
...
//- Parallel IO file handler
// uncollated (default), collated or masterUncollated
fileHandler uncollated;
//- collated: thread buffer size for queued file writes.
// If set to 0 or not sufficient for the file size threading is not used.
// Default: 2e9
maxThreadFileBufferSize 2e9;
//- masterUncollated: non-blocking buffer size.
// If the file exceeds this buffer size scheduled transfer is used.
// Default: 2e9
maxMasterFileBufferSize 2e9;
}
When using the collated file handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory in bytes that
is allocated. If the data exceeds this size, the write does not use threading.
When using the masterUncollated file handling, non-blocking MPI communication
requires a sufficiently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size in bytes of the buffer. If the
data exceeds this size, the system uses scheduled communication.
The installation defaults for the fileHandler choice, maxThreadFileBufferSize
and maxMasterFileBufferSize (set in etc/controlDict) can be over-ridden within
the case controlDict file, like other parameters. Additionally the fileHandler
can be set by:
- the "-fileHandler" command line argument;
- a FOAM_FILEHANDLER environment variable.
A foamFormatConvert utility allows users to convert files between the collated
and uncollated formats, e.g.
mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated
An example case demonstrating the file handling methods is provided in:
$FOAM_TUTORIALS/IO/fileHandling
The work was undertaken by Mattijs Janssens, in collaboration with Henry Weller.
splitMeshRegions: handle flipping of faces for surface fields
subsetMesh: subset dimensionedFields
decomposePar: use run-time selection of decomposition constraints. Used to
keep cells on particular processors. See the decomposeParDict in
$FOAM_UTILITIES/parallel/decomposePar:
- preserveBaffles: keep baffle faces on same processor
- preserveFaceZones: keep faceZones owner and neighbour on same processor
- preservePatches: keep owner and neighbour on same processor. Note: not
suitable for cyclicAMI since these are not coupled on the patch level
- singleProcessorFaceSets: keep complete faceSet on a single processor
- refinementHistory: keep cells originating from a single cell on the
same processor.
decomposePar: clean up decomposition of refinement data from snappyHexMesh
reconstructPar: reconstruct refinement data (refineHexMesh, snappyHexMesh)
reconstructParMesh: reconstruct refinement data (refineHexMesh, snappyHexMesh)
redistributePar:
- corrected mapping surfaceFields
- adding processor patches in order consistent with decomposePar
argList: check that slaves are running same version as master
fvMeshSubset: move to dynamicMesh library
fvMeshDistribute:
- support for mapping dimensionedFields
- corrected mapping of surfaceFields
parallel routines: allow parallel running on single processor
Field: support for
- distributed mapping
- mapping with flipping
mapDistribute: support for flipping
AMIInterpolation: avoid constructing localPoints
