Improvements to existing functionality
--------------------------------------
- MPI is initialised without thread support if it is not needed e.g. uncollated
- Use native c++11 threading; avoids problem with static destruction order.
- etc/cellModels now only read if needed.
- etc/controlDict can now be read from the environment variable FOAM_CONTROLDICT
- Uniform files (e.g. '0/uniform/time') are now read only once on the master only
(with the masterUncollated or collated file handlers)
- collated format writes to 'processorsNNN' instead of 'processors'. The file
format is unchanged.
- Thread buffer and file buffer size are no longer limited to 2Gb.
The global controlDict file contains parameters for file handling. Under some
circumstances, e.g. running in parallel on a system without NFS, the user may
need to set some parameters, e.g. fileHandler, before the global controlDict
file is read from file. To support this, OpenFOAM now allows the global
controlDict to be read as a string set to the FOAM_CONTROLDICT environment
variable.
The FOAM_CONTROLDICT environment variable can be set to the content the global
controlDict file, e.g. from a sh/bash shell:
export FOAM_CONTROLDICT=$(foamDictionary $FOAM_ETC/controlDict)
FOAM_CONTROLDICT can then be passed to mpirun using the -x option, e.g.:
mpirun -np 2 -x FOAM_CONTROLDICT simpleFoam -parallel
Note that while this avoids the need for NFS to read the OpenFOAM configuration
the executable still needs to load shared libraries which must either be copied
locally or available via NFS or equivalent.
New: Multiple IO ranks
----------------------
The masterUncollated and collated fileHandlers can now use multiple ranks for
writing e.g.:
mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)'
In this example ranks 0 ('processor0') and 3 ('processor3') now handle all the
I/O. Rank 0 handles 0,1,2 and rank 3 handles 3,4,5. The set of IO ranks should always
include 0 as first element and be sorted in increasing order.
The collated fileHandler uses the directory naming processorsNNN_XXX-YYY where
NNN is the total number of processors and XXX and YYY are first and last
processor in the rank, e.g. in above example the directories would be
processors6_0-2
processors6_3-5
and each of the collated files in these contains data of the local ranks
only. The same naming also applies when e.g. running decomposePar:
decomposePar -fileHandler collated -ioRanks '(0 3)'
New: Distributed data
---------------------
The individual root directories can be placed on different hosts with different
paths if necessary. In the current framework it is necessary to specify the
root per slave process but this has been simplified with the option of specifying
the root per host with the -hostRoots command line option:
mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)' \
-hostRoots '("machineA" "/tmp/" "machineB" "/tmp")'
The hostRoots option is followed by a list of machine name + root directory, the
machine name can contain regular expressions.
New: hostCollated
-----------------
The new hostCollated fileHandler automatically sets the 'ioRanks' according to
the host name with the lowest rank e.g. to run simpleFoam on 6 processors with
ranks 0-2 on machineA and ranks 3-5 on machineB with the machines specified in
the hostfile:
mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler hostCollated
This is equivalent to
mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler collated -ioRanks '(0 3)'
This example will write directories:
processors6_0-2/
processors6_3-5/
A typical example would use distributed data e.g. no two nodes, machineA and
machineB, each with three processes:
decomposePar -fileHandler collated -case cavity
# Copy case (constant/*, system/*, processors6/) to master:
rsync -a cavity machineA:/tmp/
# Create root on slave:
ssh machineB mkdir -p /tmp/cavity
# Run
mpirun --hostfile hostfile icoFoam \
-case /tmp/cavity -parallel -fileHandler hostCollated \
-hostRoots '("machineA" "/tmp" "machineB" "/tmp")'
Contributed by Mattijs Janssens
240 lines
6.7 KiB
C++
240 lines
6.7 KiB
C++
/*---------------------------------------------------------------------------*\
|
|
========= |
|
|
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
|
|
\\ / O peration |
|
|
\\ / A nd | Copyright (C) 2017-2018 OpenFOAM Foundation
|
|
\\/ M anipulation |
|
|
-------------------------------------------------------------------------------
|
|
License
|
|
This file is part of OpenFOAM.
|
|
|
|
OpenFOAM is free software: you can redistribute it and/or modify it
|
|
under the terms of the GNU General Public License as published by
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
(at your option) any later version.
|
|
|
|
OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
|
|
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
|
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
|
|
for more details.
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
Class
|
|
Foam::decomposedBlockData
|
|
|
|
Description
|
|
decomposedBlockData is a List<char> with IO on the master processor only.
|
|
|
|
SourceFiles
|
|
decomposedBlockData.C
|
|
|
|
\*---------------------------------------------------------------------------*/
|
|
|
|
#ifndef decomposedBlockData_H
|
|
#define decomposedBlockData_H
|
|
|
|
#include "IOList.H"
|
|
#include "regIOobject.H"
|
|
#include "UPstream.H"
|
|
|
|
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
|
|
|
|
namespace Foam
|
|
{
|
|
|
|
/*---------------------------------------------------------------------------*\
|
|
Class decomposedBlockData Declaration
|
|
\*---------------------------------------------------------------------------*/
|
|
|
|
class decomposedBlockData
|
|
:
|
|
public regIOobject,
|
|
public List<char>
|
|
{
|
|
protected:
|
|
|
|
// Protected data
|
|
|
|
//- Type to use for gather
|
|
const UPstream::commsTypes commsType_;
|
|
|
|
//- Communicator for all parallel comms
|
|
const label comm_;
|
|
|
|
|
|
// Protected member functions
|
|
|
|
//- Helper: determine number of processors whose recvSizes fits
|
|
// ito maxBufferSize
|
|
static label calcNumProcs
|
|
(
|
|
const label comm,
|
|
const off_t maxBufferSize,
|
|
const labelUList& recvSizes,
|
|
const label startProci
|
|
);
|
|
|
|
//- Read data into *this. ISstream is only valid on master.
|
|
static bool readBlocks
|
|
(
|
|
const label comm,
|
|
autoPtr<ISstream>& isPtr,
|
|
List<char>& data,
|
|
const UPstream::commsTypes commsType
|
|
);
|
|
|
|
|
|
public:
|
|
|
|
TypeName("decomposedBlockData");
|
|
|
|
|
|
// Constructors
|
|
|
|
//- Construct given an IOobject
|
|
decomposedBlockData
|
|
(
|
|
const label comm,
|
|
const IOobject&,
|
|
const UPstream::commsTypes = UPstream::commsTypes::scheduled
|
|
);
|
|
|
|
//- Construct given an IOobject and for READ_IF_MODIFIED a List<char>
|
|
decomposedBlockData
|
|
(
|
|
const label comm,
|
|
const IOobject&,
|
|
const UList<char>&,
|
|
const UPstream::commsTypes = UPstream::commsTypes::scheduled
|
|
);
|
|
|
|
//- Construct by transferring the IOList contents
|
|
decomposedBlockData
|
|
(
|
|
const label comm,
|
|
const IOobject&,
|
|
const Xfer<List<char>>&,
|
|
const UPstream::commsTypes = UPstream::commsTypes::scheduled
|
|
);
|
|
|
|
|
|
//- Destructor
|
|
virtual ~decomposedBlockData();
|
|
|
|
|
|
// Member functions
|
|
|
|
//- Read object
|
|
virtual bool read();
|
|
|
|
//- Write separated content. Assumes content is the serialised data
|
|
// and that the master data contains a header
|
|
virtual bool writeData(Ostream&) const;
|
|
|
|
//- Write using given format, version and compression
|
|
virtual bool writeObject
|
|
(
|
|
IOstream::streamFormat,
|
|
IOstream::versionNumber,
|
|
IOstream::compressionType,
|
|
const bool valid
|
|
) const;
|
|
|
|
|
|
// Helpers
|
|
|
|
//- Read header. Call only on master.
|
|
static bool readMasterHeader(IOobject&, Istream&);
|
|
|
|
//- Helper: write FoamFile IOobject header
|
|
static void writeHeader
|
|
(
|
|
Ostream& os,
|
|
const IOstream::versionNumber version,
|
|
const IOstream::streamFormat format,
|
|
const word& type,
|
|
const string& note,
|
|
const fileName& location,
|
|
const word& name
|
|
);
|
|
|
|
//- Read selected block (non-seeking) + header information
|
|
static autoPtr<ISstream> readBlock
|
|
(
|
|
const label blocki,
|
|
Istream& is,
|
|
IOobject& headerIO
|
|
);
|
|
|
|
//- Read master header information (into headerIO) and return
|
|
// data in stream. Note: isPtr is only valid on master.
|
|
static autoPtr<ISstream> readBlocks
|
|
(
|
|
const label comm,
|
|
const fileName& fName,
|
|
autoPtr<ISstream>& isPtr,
|
|
IOobject& headerIO,
|
|
const UPstream::commsTypes commsType
|
|
);
|
|
|
|
//- Helper: gather single label. Note: using native Pstream.
|
|
// datas sized with num procs but undefined contents on
|
|
// slaves
|
|
static void gather
|
|
(
|
|
const label comm,
|
|
const label data,
|
|
labelList& datas
|
|
);
|
|
|
|
//- Helper: gather data from (subset of) slaves. Returns
|
|
// recvData : received data
|
|
// recvOffsets : offset in data. recvOffsets is nProcs+1
|
|
static void gatherSlaveData
|
|
(
|
|
const label comm,
|
|
const UList<char>& data,
|
|
const labelUList& recvSizes,
|
|
|
|
const label startProc,
|
|
const label nProcs,
|
|
|
|
List<int>& recvOffsets,
|
|
List<char>& recvData
|
|
);
|
|
|
|
//- Write *this. Ostream only valid on master. Returns starts of
|
|
// processor blocks
|
|
static bool writeBlocks
|
|
(
|
|
const label comm,
|
|
autoPtr<OSstream>& osPtr,
|
|
List<std::streamoff>& start,
|
|
const UList<char>& masterData,
|
|
|
|
const labelUList& recvSizes,
|
|
|
|
// optional slave data (on master)
|
|
const PtrList<SubList<char>>& slaveData,
|
|
|
|
const UPstream::commsTypes,
|
|
const bool syncReturnState = true
|
|
);
|
|
|
|
//- Detect number of blocks in a file
|
|
static label numBlocks(const fileName&);
|
|
};
|
|
|
|
|
|
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
|
|
|
|
} // End namespace Foam
|
|
|
|
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
|
|
|
|
#endif
|
|
|
|
// ************************************************************************* //
|