DecomposePar and reconstructPar now interleave the processing of
multiple regions. This means that with the -allRegions option, the
earlier times are completed in their entirety before later times are
considered. It also lets regions to access each other during
decomposition and reconstruction, which will be important for
non-conformal region interfaces.
To aid interpretation of the log, region prefixing is now used by both
utilities in the same way as is done by foamMultiRun.
DecomposePar has been overhauled so that it matches reconstructPar much
more closely, both in terms of output and of iteration sequence. All
meshes and addressing are loaded simultaneously and each field is
considered in turn. Previously, all the fields were loaded, and each
process and addressing set was considered in turn. This new strategy
optimises memory usage for cases with lots of fields.
The timeName() function simply returns the dimensionedScalar::name() which holds
the user-time name of the current time and now that timeName() is no longer
virtual the dimensionedScalar::name() can be called directly. The timeName()
function implementation is maintained for backward-compatibility.
This reference represents unnecessary storage. The mesh can be obtained
from tracking data or passed to the particle evolution functions by
argument.
In addition, removing the mesh reference makes it possible to construct
as particle from an Istream without the need for an iNew class. This
simplifies stream-based transfer, and makes it possible for particles to
be communicated by a polyDistributionMap.
in which different solver modules can be selected in each region to for complex
conjugate heat-transfer and other combined physics problems such as FSI
(fluid-structure interaction).
For single-region simulations the solver module is selected, instantiated and
executed in the PIMPLE loop in the new foamRun application.
For multi-region simulations the set of solver modules, one for each region, are
selected, instantiated and executed in the multi-region PIMPLE loop of new the
foamMultiRun application.
This provides a very general, flexible and extensible framework for complex
coupled problems by creating more solver modules, either by converting existing
solver applications or creating new ones.
The current set of solver modules provided are:
isothermalFluid
Solver module for steady or transient turbulent flow of compressible
isothermal fluids with optional mesh motion and mesh topology changes.
Created from the rhoSimpleFoam, rhoPimpleFoam and buoyantFoam solvers but
without the energy equation, hence isothermal. The buoyant pressure
formulation corresponding to the buoyantFoam solver is selected
automatically by the presence of the p_rgh pressure field in the start-time
directory.
fluid
Solver module for steady or transient turbulent flow of compressible fluids
with heat-transfer for HVAC and similar applications, with optional
mesh motion and mesh topology changes.
Derived from the isothermalFluid solver module with the addition of the
energy equation from the rhoSimpleFoam, rhoPimpleFoam and buoyantFoam
solvers, thus providing the equivalent functionality of these three solvers.
multicomponentFluid
Solver module for steady or transient turbulent flow of compressible
reacting fluids with optional mesh motion and mesh topology changes.
Derived from the isothermalFluid solver module with the addition of
multicomponent thermophysical properties energy and specie mass-fraction
equations from the reactingFoam solver, thus providing the equivalent
functionality in reactingFoam and buoyantReactingFoam. Chemical reactions
and/or combustion modelling may be optionally selected to simulate reacting
systems including fires, explosions etc.
solid
Solver module for turbulent flow of compressible fluids for conjugate heat
transfer, HVAC and similar applications, with optional mesh motion and mesh
topology changes.
The solid solver module may be selected in solid regions of a CHT case, with
either the fluid or multicomponentFluid solver module in the fluid regions
and executed with foamMultiRun to provide functionality equivalent
chtMultiRegionFoam but in a flexible and extensible framework for future
extension to more complex coupled problems.
All the usual fvModels, fvConstraints, functionObjects etc. are available with
these solver modules to support simulations including body-forces, local sources,
Lagrangian clouds, liquid films etc. etc.
Converting compressibleInterFoam and multiphaseEulerFoam into solver modules
would provide a significant enhancement to the CHT capability and incompressible
solvers like pimpleFoam run in conjunction with solidDisplacementFoam in
foamMultiRun would be useful for a range of FSI problems. Many other
combinations of existing solvers converted into solver modules could prove
useful for a very wide range of complex combined physics simulations.
All tutorials from the rhoSimpleFoam, rhoPimpleFoam, buoyantFoam, reactingFoam,
buoyantReactingFoam and chtMultiRegionFoam solver applications replaced by
solver modules have been updated and moved into the tutorials/modules directory:
modules
├── CHT
│ ├── coolingCylinder2D
│ ├── coolingSphere
│ ├── heatedDuct
│ ├── heatExchanger
│ ├── reverseBurner
│ └── shellAndTubeHeatExchanger
├── fluid
│ ├── aerofoilNACA0012
│ ├── aerofoilNACA0012Steady
│ ├── angledDuct
│ ├── angledDuctExplicitFixedCoeff
│ ├── angledDuctLTS
│ ├── annularThermalMixer
│ ├── BernardCells
│ ├── blockedChannel
│ ├── buoyantCavity
│ ├── cavity
│ ├── circuitBoardCooling
│ ├── decompressionTank
│ ├── externalCoupledCavity
│ ├── forwardStep
│ ├── helmholtzResonance
│ ├── hotRadiationRoom
│ ├── hotRadiationRoomFvDOM
│ ├── hotRoom
│ ├── hotRoomBoussinesq
│ ├── hotRoomBoussinesqSteady
│ ├── hotRoomComfort
│ ├── iglooWithFridges
│ ├── mixerVessel2DMRF
│ ├── nacaAirfoil
│ ├── pitzDaily
│ ├── prism
│ ├── shockTube
│ ├── squareBend
│ ├── squareBendLiq
│ └── squareBendLiqSteady
└── multicomponentFluid
├── aachenBomb
├── counterFlowFlame2D
├── counterFlowFlame2D_GRI
├── counterFlowFlame2D_GRI_TDAC
├── counterFlowFlame2DLTS
├── counterFlowFlame2DLTS_GRI_TDAC
├── cylinder
├── DLR_A_LTS
├── filter
├── hotBoxes
├── membrane
├── parcelInBox
├── rivuletPanel
├── SandiaD_LTS
├── simplifiedSiwek
├── smallPoolFire2D
├── smallPoolFire3D
├── splashPanel
├── verticalChannel
├── verticalChannelLTS
└── verticalChannelSteady
Also redirection scripts are provided for the replaced solvers which call
foamRun -solver <solver module name> or foamMultiRun in the case of
chtMultiRegionFoam for backward-compatibility.
Documentation for foamRun and foamMultiRun:
Application
foamRun
Description
Loads and executes an OpenFOAM solver module either specified by the
optional \c solver entry in the \c controlDict or as a command-line
argument.
Uses the flexible PIMPLE (PISO-SIMPLE) solution for time-resolved and
pseudo-transient and steady simulations.
Usage
\b foamRun [OPTION]
- \par -solver <name>
Solver name
- \par -libs '(\"lib1.so\" ... \"libN.so\")'
Specify the additional libraries loaded
Example usage:
- To run a \c rhoPimpleFoam case by specifying the solver on the
command line:
\verbatim
foamRun -solver fluid
\endverbatim
- To update and run a \c rhoPimpleFoam case add the following entries to
the controlDict:
\verbatim
application foamRun;
solver fluid;
\endverbatim
then execute \c foamRun
Application
foamMultiRun
Description
Loads and executes an OpenFOAM solver modules for each region of a
multiregion simulation e.g. for conjugate heat transfer.
The region solvers are specified in the \c regionSolvers dictionary entry in
\c controlDict, containing a list of pairs of region and solver names,
e.g. for a two region case with one fluid region named
liquid and one solid region named tubeWall:
\verbatim
regionSolvers
{
liquid fluid;
tubeWall solid;
}
\endverbatim
The \c regionSolvers entry is a dictionary to support name substitutions to
simplify the specification of a single solver type for a set of
regions, e.g.
\verbatim
fluidSolver fluid;
solidSolver solid;
regionSolvers
{
tube1 $fluidSolver;
tubeWall1 solid;
tube2 $fluidSolver;
tubeWall2 solid;
tube3 $fluidSolver;
tubeWall3 solid;
}
\endverbatim
Uses the flexible PIMPLE (PISO-SIMPLE) solution for time-resolved and
pseudo-transient and steady simulations.
Usage
\b foamMultiRun [OPTION]
- \par -libs '(\"lib1.so\" ... \"libN.so\")'
Specify the additional libraries loaded
Example usage:
- To update and run a \c chtMultiRegion case add the following entries to
the controlDict:
\verbatim
application foamMultiRun;
regionSolvers
{
fluid fluid;
solid solid;
}
\endverbatim
then execute \c foamMultiRun
The cellProc field is the field of cell-processor labels.
The names "distribution" and "dist" have been removed as these are
ambiguous in relation to other forms of distribution and to distance.
The reconstructPar utility now reconstructs the mesh if and when it is
necessary to do so. The reconstructParMesh utility is therefore no
longer necessary and has been removed.
It was necessary/advantagous to consolidate these utilities into one
because in the case of mesh changes it becomes increasingly less clear
which of the separate utilities is responsible for reconstructing data
that is neither clearly physical field nor mesh topology; e.g., moving
points, sets, refinement data, and so on.
This major development provides coupling of patches which are
non-conformal, i.e. where the faces of one patch do not match the faces
of the other. The coupling is fully conservative and second order
accurate in space, unlike the Arbitrary Mesh Interface (AMI) and
associated ACMI and Repeat AMI methods which NCC replaces.
Description:
A non-conformal couple is a connection between a pair of boundary
patches formed by projecting one patch onto the other in a way that
fills the space between them. The intersection between the projected
surface and patch forms new faces that are incorporated into the finite
volume mesh. These new faces are created identically on both sides of
the couple, and therefore become equivalent to internal faces within the
mesh. The affected cells remain closed, meaning that the area vectors
sum to zero for all the faces of each cell. Consequently, the main
benefits of the finite volume method, i.e. conservation and accuracy,
are not undermined by the coupling.
A couple connects parts of mesh that are otherwise disconnected and can
be used in the following ways:
+ to simulate rotating geometries, e.g. a propeller or stirrer, in which
a part of the mesh rotates with the geometry and connects to a
surrounding mesh which is not moving;
+ to connect meshes that are generated separately, which do not conform
at their boundaries;
+ to connect patches which only partially overlap, in which the
non-overlapped section forms another boundary, e.g. a wall;
+ to simulate a case with a geometry which is periodically repeating by
creating multiple couples with different transformations between
patches.
The capability for simulating partial overlaps replaces the ACMI
functionality, currently provided by the 'cyclicACMI' patch type, and
which is unreliable unless the couple is perfectly flat. The capability
for simulating periodically repeating geometry replaces the Repeat AMI
functionality currently provided by the 'cyclicRepeatAMI' patch type.
Usage:
The process of meshing for NCC is very similar to existing processes for
meshing for AMI. Typically, a mesh is generated with an identifiable set
of internal faces which coincide with the surface through which the mesh
will be coupled. These faces are then duplicated by running the
'createBaffles' utility to create two boundary patches. The points are
then split using 'splitBaffles' in order to permit independent motion of
the patches.
In AMI, these patches are assigned the 'cyclicAMI' patch type, which
couples them using AMI interpolation methods.
With NCC, the patches remain non-coupled, e.g. a 'wall' type. Coupling
is instead achieved by running the new 'createNonConformalCouples'
utility, which creates additional coupled patches of type
'nonConformalCyclic'. These appear in the 'constant/polyMesh/boundary'
file with zero faces; they are populated with faces in the finite volume
mesh during the connection process in NCC.
For a single couple, such as that which separates the rotating and
stationary sections of a mesh, the utility can be called using the
non-coupled patch names as arguments, e.g.
createNonConformalCouples -overwrite rotatingZoneInner rotatingZoneOuter
where 'rotatingZoneInner' and 'rotatingZoneOuter' are the names of the
patches.
For multiple couples, and/or couples with transformations,
'createNonConformalCouples' should be run without arguments. Settings
will then be read from a configuration file named
'system/createNonConformalCouplesDict'. See
'$FOAM_ETC/caseDicts/annotated/createNonConformalCouplesDict' for
examples.
Boundary conditions must be specified for the non-coupled patches. For a
couple where the patches fully overlap, boundary conditions
corresponding to a slip wall are typically applied to fields, i.e
'movingWallSlipVelocity' (or 'slip' if the mesh is stationary) for
velocity U, 'zeroGradient' or 'fixedFluxPressure' for pressure p, and
'zeroGradient' for other fields. For a couple with
partially-overlapping patches, boundary conditions are applied which
physically represent the non-overlapped region, e.g. a no-slip wall.
Boundary conditions also need to be specified for the
'nonConformalCyclic' patches created by 'createNonConformalCouples'. It
is generally recommended that this is done by including the
'$FOAM_ETC/caseDicts/setConstraintTypes' file in the 'boundaryField'
section of each of the field files, e.g.
boundaryField
{
#includeEtc "caseDicts/setConstraintTypes"
inlet
{
...
}
...
}
For moving mesh cases, it may be necessary to correct the mesh fluxes
that are changed as a result of the connection procedure. If the
connected patches do not conform perfectly to the mesh motion, then
failure to correct the fluxes can result in noise in the pressure
solution.
Correction for the mesh fluxes is enabled by the 'correctMeshPhi' switch
in the 'PIMPLE' (or equivalent) section of 'system/fvSolution'. When it
is enabled, solver settings are required for 'MeshPhi'. The solution
just needs to distribute the error enough to dissipate the noise. A
smooth solver with a loose tolerance is typically sufficient, e.g. the
settings in 'system/fvSolution' shown below:
solvers
{
MeshPhi
{
solver smoothSolver;
smoother symGaussSeidel;
tolerance 1e-2;
relTol 0;
}
...
}
PIMPLE
{
correctMeshPhi yes;
...
}
The solution of 'MeshPhi' is an inexpensive computation since it is
applied only to a small subset of the mesh adjacent to the
couple. Conservation is maintained whether or not the mesh flux
correction is enabled, and regardless of the solution tolerance for
'MeshPhi'.
Advantages of NCC:
+ NCC maintains conservation which is required for many numerical
schemes and algorithms to operate effectively, in particular those
designed to maintain boundedness of a solution.
+ Closed-volume systems no longer suffer from accumulation or loss of
mass, poor convergence of the pressure equation, and/or concentration
of error in the reference cell.
+ Partially overlapped simulations are now possible on surfaces that are
not perfectly flat. The projection fills space so no overlaps or
spaces are generated inside contiguously overlapping sections, even if
those sections have sharp angles.
+ The finite volume faces created by NCC have geometrically accurate
centres. This makes the method second order accurate in space.
+ The polyhedral mesh no longer requires duplicate boundary faces to be
generated in order to run a partially overlapped simulation.
+ Lagrangian elements can now transfer across non-conformal couplings in
parallel.
+ Once the intersection has been computed and applied to the finite
volume mesh, it can use standard cyclic or processor cyclic finite
volume boundary conditions, with no need for additional patch types or
matrix interfaces.
+ Parallel communication is done using the standard
processor-patch-field system. This is more efficient than alternative
systems since it has been carefully optimised for use within the
linear solvers.
+ Coupled patches are disconnected prior to mesh motion and topology
change and reconnected afterwards. This simplifies the boundary
condition specification for mesh motion fields.
Resolved Bug Reports:
+ https://bugs.openfoam.org/view.php?id=663
+ https://bugs.openfoam.org/view.php?id=883
+ https://bugs.openfoam.org/view.php?id=887
+ https://bugs.openfoam.org/view.php?id=1337
+ https://bugs.openfoam.org/view.php?id=1388
+ https://bugs.openfoam.org/view.php?id=1422
+ https://bugs.openfoam.org/view.php?id=1829
+ https://bugs.openfoam.org/view.php?id=1841
+ https://bugs.openfoam.org/view.php?id=2274
+ https://bugs.openfoam.org/view.php?id=2561
+ https://bugs.openfoam.org/view.php?id=3817
Deprecation:
NCC replaces the functionality provided by AMI, ACMI and Repeat AMI.
ACMI and Repeat AMI are insufficiently reliable to warrant further
maintenance so are removed in an accompanying commit to OpenFOAM-dev.
AMI is more widely used so will be retained alongside NCC for the next
version release of OpenFOAM and then subsequently removed from
OpenFOAM-dev.
boundaryProcAddressing has been removed. This has not been needed for a
long time. decomposePar has been optimised for mininum IO, rather than
minimum memory usage. decomposePar has also been corrected so that it
can decompose sequences of time-varying meshes.
When snappyHexMesh is run in parallel it re-balances the mesh during refinement
and layer addition by redistribution which requires a decomposition method
that operates in parallel, e.g. hierachical or ptscotch. decomposePar uses a
decomposition method which operates in serial e.g. hierachical but NOT
ptscotch. In order to run decomposePar followed by snappyHexMesh in parallel it
has been necessary to change the method specified in decomposeParDict but now
this is avoided by separately specifying the decomposition and distribution
methods, e.g. in the incompressible/simpleFoam/motorBike case:
numberOfSubdomains 6;
decomposer hierarchical;
distributor ptscotch;
hierarchicalCoeffs
{
n (3 2 1);
order xyz;
}
The distributor entry is also used for run-time mesh redistribution, e.g. in the
multiphase/interFoam/RAS/floatingObject case re-distribution for load-balancing
is enabled in constant/dynamicMeshDict:
distributor
{
type distributor;
libs ("libfvMeshDistributors.so");
redistributionInterval 10;
}
which uses the distributor specified in system/decomposeParDict:
distributor hierarchical;
This rationalisation provides the structure for development of mesh
redistribution and load-balancing.
replacing the virtual functions overridden in engineTime.
Now the userTime conversion function in Time is specified in system/controlDict
such that the solver as well as all pre- and post-processing tools also operate
correctly with the chosen user-time.
For example the user-time and rpm in the tutorials/combustion/XiEngineFoam/kivaTest case are
now specified in system/controlDict:
userTime
{
type engine;
rpm 1500;
}
The default specification is real-time:
userTime
{
type real;
}
but this entry can be omitted as the real-time class is instantiated
automatically if the userTime entry is not present in system/controlDict.
Mesh motion and topology change are now combinable run-time selectable options
within fvMesh, replacing the restrictive dynamicFvMesh which supported only
motion OR topology change.
All solvers which instantiated a dynamicFvMesh now instantiate an fvMesh which
reads the optional constant/dynamicFvMeshDict to construct an fvMeshMover and/or
an fvMeshTopoChanger. These two are specified within the optional mover and
topoChanger sub-dictionaries of dynamicFvMeshDict.
When the fvMesh is updated the fvMeshTopoChanger is first executed which can
change the mesh topology in anyway, adding or removing points as required, for
example for automatic mesh refinement/unrefinement, and all registered fields
are mapped onto the updated mesh. The fvMeshMover is then executed which moved
the points only and calculates the cell volume change and corresponding
mesh-fluxes for conservative moving mesh transport. If multiple topological
changes or movements are required these would be combined into special
fvMeshMovers and fvMeshTopoChangers which handle the processing of a list of
changes, e.g. solidBodyMotionFunctions:multiMotion.
The tutorials/multiphase/interFoam/laminar/sloshingTank3D3DoF case has been
updated to demonstrate this new functionality by combining solid-body motion
with mesh refinement/unrefinement:
/*--------------------------------*- C++ -*----------------------------------*\
========= |
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\\ / O peration | Website: https://openfoam.org
\\ / A nd | Version: dev
\\/ M anipulation |
\*---------------------------------------------------------------------------*/
FoamFile
{
format ascii;
class dictionary;
location "constant";
object dynamicMeshDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
mover
{
type motionSolver;
libs ("libfvMeshMovers.so" "libfvMotionSolvers.so");
motionSolver solidBody;
solidBodyMotionFunction SDA;
CofG (0 0 0);
lamda 50;
rollAmax 0.2;
rollAmin 0.1;
heaveA 4;
swayA 2.4;
Q 2;
Tp 14;
Tpn 12;
dTi 0.06;
dTp -0.001;
}
topoChanger
{
type refiner;
libs ("libfvMeshTopoChangers.so");
// How often to refine
refineInterval 1;
// Field to be refinement on
field alpha.water;
// Refine field in between lower..upper
lowerRefineLevel 0.001;
upperRefineLevel 0.999;
// Have slower than 2:1 refinement
nBufferLayers 1;
// Refine cells only up to maxRefinement levels
maxRefinement 1;
// Stop refinement if maxCells reached
maxCells 200000;
// Flux field and corresponding velocity field. Fluxes on changed
// faces get recalculated by interpolating the velocity. Use 'none'
// on surfaceScalarFields that do not need to be reinterpolated.
correctFluxes
(
(phi none)
(nHatf none)
(rhoPhi none)
(alphaPhi.water none)
(meshPhi none)
(meshPhi_0 none)
(ghf none)
);
// Write the refinement level as a volScalarField
dumpLevel true;
}
// ************************************************************************* //
Note that currently this is the only working combination of mesh-motion with
topology change within the new framework and further development is required to
update the set of topology changers so that topology changes with mapping are
separated from the mesh-motion so that they can be combined with any of the
other movements or topology changes in any manner.
All of the solvers and tutorials have been updated to use the new form of
dynamicMeshDict but backward-compatibility was not practical due to the complete
reorganisation of the mesh change structure.
used to check the existence of and open an object file, read and check the
header without constructing the object.
'typeIOobject' operates in an equivalent and consistent manner to 'regIOobject'
but the type information is provided by the template argument rather than via
virtual functions for which the derived object would need to be constructed,
which is the case for 'regIOobject'.
'typeIOobject' replaces the previous separate functions 'typeHeaderOk' and
'typeFilePath' with a single consistent interface.
now all path functions in 'IOobject' are either templated on the type or require a
'globalFile' argument to specify if the type is case global e.g. 'IOdictionary' or
decomposed in parallel, e.g. almost everything else.
The 'global()' and 'globalFile()' virtual functions are now in 'regIOobject'
abstract base-class and overridden as required by derived classes. The path
functions using 'global()' and 'globalFile()' to differentiate between global
and processor local objects are now also in 'regIOobject' rather than 'IOobject'
to ensure the path returned is absolutely consistent with the type.
Unfortunately there is still potential for unexpected IO behaviour inconsistent
with the global/local nature of the type due to the 'fileOperation' classes
searching the processor directory for case global objects before searching the
case directory. This approach appears to be a work-around for incomplete
integration with and rationalisation of 'IOobject' but with the changes above it
is no longer necessary. Unfortunately this "up" searching is baked-in at a low
level and mixed-up with various complex ways to pick the processor directory
name out of the object path and will take some unravelling but this work will
undertaken as time allows.
The -dict option is now handled correctly and consistently across all
applications with -dict options. The logic associated with doing so has
been centralised.
If a relative path is given to the -dict option, then it is assumed to
be relative to the case directory. If an absolute path is given, then it
is used without reference to the case directory. In both cases, if the
path is found to be a directory, then the standard dictionary name is
appended to the path.
Resolves bug report http://bugs.openfoam.org/view.php?id=3692
For many information and diagnostic messages the absolute path of the object is
not required and the local path relative to the current case is sufficient; the
new localObjectPath() member function of IOobject provides a convenient way of
printing this.
Registration occurs when the temporary field is transferred to a non-temporary
field via a constructor or if explicitly transferred to the database via the
regIOobject "store" methods.
for
db/functionObjects/timeControl/timeControl.H: timeControls
primitives/bools/Switch/Switch.H: class switchType
primitives/strings/fileName/fileName.H: fileType
primitives/strings/wordRe/wordRe.H: compOption
This method waits until all the threads have completed IO operations and
then clears any cached information about the files on disk. This
replaces the deactivation of threading by means of zeroing the buffer
size when writing and reading of a file happen in sequence. It also
allows paraFoam to update the list of available times.
Patch contributed by Mattijs Janssens
Resolves bug report https://bugs.openfoam.org/view.php?id=2962
Improvements to existing functionality
--------------------------------------
- MPI is initialised without thread support if it is not needed e.g. uncollated
- Use native c++11 threading; avoids problem with static destruction order.
- etc/cellModels now only read if needed.
- etc/controlDict can now be read from the environment variable FOAM_CONTROLDICT
- Uniform files (e.g. '0/uniform/time') are now read only once on the master only
(with the masterUncollated or collated file handlers)
- collated format writes to 'processorsNNN' instead of 'processors'. The file
format is unchanged.
- Thread buffer and file buffer size are no longer limited to 2Gb.
The global controlDict file contains parameters for file handling. Under some
circumstances, e.g. running in parallel on a system without NFS, the user may
need to set some parameters, e.g. fileHandler, before the global controlDict
file is read from file. To support this, OpenFOAM now allows the global
controlDict to be read as a string set to the FOAM_CONTROLDICT environment
variable.
The FOAM_CONTROLDICT environment variable can be set to the content the global
controlDict file, e.g. from a sh/bash shell:
export FOAM_CONTROLDICT=$(foamDictionary $FOAM_ETC/controlDict)
FOAM_CONTROLDICT can then be passed to mpirun using the -x option, e.g.:
mpirun -np 2 -x FOAM_CONTROLDICT simpleFoam -parallel
Note that while this avoids the need for NFS to read the OpenFOAM configuration
the executable still needs to load shared libraries which must either be copied
locally or available via NFS or equivalent.
New: Multiple IO ranks
----------------------
The masterUncollated and collated fileHandlers can now use multiple ranks for
writing e.g.:
mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)'
In this example ranks 0 ('processor0') and 3 ('processor3') now handle all the
I/O. Rank 0 handles 0,1,2 and rank 3 handles 3,4,5. The set of IO ranks should always
include 0 as first element and be sorted in increasing order.
The collated fileHandler uses the directory naming processorsNNN_XXX-YYY where
NNN is the total number of processors and XXX and YYY are first and last
processor in the rank, e.g. in above example the directories would be
processors6_0-2
processors6_3-5
and each of the collated files in these contains data of the local ranks
only. The same naming also applies when e.g. running decomposePar:
decomposePar -fileHandler collated -ioRanks '(0 3)'
New: Distributed data
---------------------
The individual root directories can be placed on different hosts with different
paths if necessary. In the current framework it is necessary to specify the
root per slave process but this has been simplified with the option of specifying
the root per host with the -hostRoots command line option:
mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)' \
-hostRoots '("machineA" "/tmp/" "machineB" "/tmp")'
The hostRoots option is followed by a list of machine name + root directory, the
machine name can contain regular expressions.
New: hostCollated
-----------------
The new hostCollated fileHandler automatically sets the 'ioRanks' according to
the host name with the lowest rank e.g. to run simpleFoam on 6 processors with
ranks 0-2 on machineA and ranks 3-5 on machineB with the machines specified in
the hostfile:
mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler hostCollated
This is equivalent to
mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler collated -ioRanks '(0 3)'
This example will write directories:
processors6_0-2/
processors6_3-5/
A typical example would use distributed data e.g. no two nodes, machineA and
machineB, each with three processes:
decomposePar -fileHandler collated -case cavity
# Copy case (constant/*, system/*, processors6/) to master:
rsync -a cavity machineA:/tmp/
# Create root on slave:
ssh machineB mkdir -p /tmp/cavity
# Run
mpirun --hostfile hostfile icoFoam \
-case /tmp/cavity -parallel -fileHandler hostCollated \
-hostRoots '("machineA" "/tmp" "machineB" "/tmp")'
Contributed by Mattijs Janssens
When an OpenFOAM simulation runs in parallel, the data for decomposed fields and
mesh(es) has historically been stored in multiple files within separate
directories for each processor. Processor directories are named 'processorN',
where N is the processor number.
This commit introduces an alternative "collated" file format where the data for
each decomposed field (and mesh) is collated into a single file, which is
written and read on the master processor. The files are stored in a single
directory named 'processors'.
The new format produces significantly fewer files - one per field, instead of N
per field. For large parallel cases, this avoids the restriction on the number
of open files imposed by the operating system limits.
The file writing can be threaded allowing the simulation to continue running
while the data is being written to file. NFS (Network File System) is not
needed when using the the collated format and additionally, there is an option
to run without NFS with the original uncollated approach, known as
"masterUncollated".
The controls for the file handling are in the OptimisationSwitches of
etc/controlDict:
OptimisationSwitches
{
...
//- Parallel IO file handler
// uncollated (default), collated or masterUncollated
fileHandler uncollated;
//- collated: thread buffer size for queued file writes.
// If set to 0 or not sufficient for the file size threading is not used.
// Default: 2e9
maxThreadFileBufferSize 2e9;
//- masterUncollated: non-blocking buffer size.
// If the file exceeds this buffer size scheduled transfer is used.
// Default: 2e9
maxMasterFileBufferSize 2e9;
}
When using the collated file handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory in bytes that
is allocated. If the data exceeds this size, the write does not use threading.
When using the masterUncollated file handling, non-blocking MPI communication
requires a sufficiently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size in bytes of the buffer. If the
data exceeds this size, the system uses scheduled communication.
The installation defaults for the fileHandler choice, maxThreadFileBufferSize
and maxMasterFileBufferSize (set in etc/controlDict) can be over-ridden within
the case controlDict file, like other parameters. Additionally the fileHandler
can be set by:
- the "-fileHandler" command line argument;
- a FOAM_FILEHANDLER environment variable.
A foamFormatConvert utility allows users to convert files between the collated
and uncollated formats, e.g.
mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated
An example case demonstrating the file handling methods is provided in:
$FOAM_TUTORIALS/IO/fileHandling
The work was undertaken by Mattijs Janssens, in collaboration with Henry Weller.
Using
decomposePar -copyZero
The mesh is decomposed as usual but the '0' directory is recursively copied to
the 'processor.*' directories rather than decomposing the fields. This is a
convenient option to handle cases where the initial field files are generic and
can be used for serial or parallel running. See for example the
incompressible/simpleFoam/motorBike tutorial case.
splitMeshRegions: handle flipping of faces for surface fields
subsetMesh: subset dimensionedFields
decomposePar: use run-time selection of decomposition constraints. Used to
keep cells on particular processors. See the decomposeParDict in
$FOAM_UTILITIES/parallel/decomposePar:
- preserveBaffles: keep baffle faces on same processor
- preserveFaceZones: keep faceZones owner and neighbour on same processor
- preservePatches: keep owner and neighbour on same processor. Note: not
suitable for cyclicAMI since these are not coupled on the patch level
- singleProcessorFaceSets: keep complete faceSet on a single processor
- refinementHistory: keep cells originating from a single cell on the
same processor.
decomposePar: clean up decomposition of refinement data from snappyHexMesh
reconstructPar: reconstruct refinement data (refineHexMesh, snappyHexMesh)
reconstructParMesh: reconstruct refinement data (refineHexMesh, snappyHexMesh)
redistributePar:
- corrected mapping surfaceFields
- adding processor patches in order consistent with decomposePar
argList: check that slaves are running same version as master
fvMeshSubset: move to dynamicMesh library
fvMeshDistribute:
- support for mapping dimensionedFields
- corrected mapping of surfaceFields
parallel routines: allow parallel running on single processor
Field: support for
- distributed mapping
- mapping with flipping
mapDistribute: support for flipping
AMIInterpolation: avoid constructing localPoints
To be used instead of zeroGradientFvPatchField for temporary fields for
which zero-gradient extrapolation is use to evaluate the boundary field
but avoiding fields derived from temporary field using field algebra
inheriting the zeroGradient boundary condition by the reuse of the
temporary field storage.
zeroGradientFvPatchField should not be used as the default patch field
for any temporary fields and should be avoided for non-temporary fields
except where it is clearly appropriate;
extrapolatedCalculatedFvPatchField and calculatedFvPatchField are
generally more suitable defaults depending on the manner in which the
boundary values are specified or evaluated.
The entire OpenFOAM-dev code-base has been updated following the above
recommendations.
Henry G. Weller
CFD Direct
