Files
OpenFOAM-12/src/OpenFOAM/db/IOobjects/decomposedBlockData/decomposedBlockData.H
Henry Weller a4de83a425 Improvements to the fileHandler and collated IO
Improvements to existing functionality
--------------------------------------
  - MPI is initialised without thread support if it is not needed e.g. uncollated
  - Use native c++11 threading; avoids problem with static destruction order.
  - etc/cellModels now only read if needed.
  - etc/controlDict can now be read from the environment variable FOAM_CONTROLDICT
  - Uniform files (e.g. '0/uniform/time') are now read only once on the master only
    (with the masterUncollated or collated file handlers)
  - collated format writes to 'processorsNNN' instead of 'processors'.  The file
    format is unchanged.
  - Thread buffer and file buffer size are no longer limited to 2Gb.

The global controlDict file contains parameters for file handling.  Under some
circumstances, e.g. running in parallel on a system without NFS, the user may
need to set some parameters, e.g. fileHandler, before the global controlDict
file is read from file.  To support this, OpenFOAM now allows the global
controlDict to be read as a string set to the FOAM_CONTROLDICT environment
variable.

The FOAM_CONTROLDICT environment variable can be set to the content the global
controlDict file, e.g. from a sh/bash shell:

    export FOAM_CONTROLDICT=$(foamDictionary $FOAM_ETC/controlDict)

FOAM_CONTROLDICT can then be passed to mpirun using the -x option, e.g.:

    mpirun -np 2 -x FOAM_CONTROLDICT simpleFoam -parallel

Note that while this avoids the need for NFS to read the OpenFOAM configuration
the executable still needs to load shared libraries which must either be copied
locally or available via NFS or equivalent.

New: Multiple IO ranks
----------------------
The masterUncollated and collated fileHandlers can now use multiple ranks for
writing e.g.:

    mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)'

In this example ranks 0 ('processor0') and 3 ('processor3') now handle all the
I/O.  Rank 0 handles 0,1,2 and rank 3 handles 3,4,5.  The set of IO ranks should always
include 0 as first element and be sorted in increasing order.

The collated fileHandler uses the directory naming processorsNNN_XXX-YYY where
NNN is the total number of processors and XXX and YYY are first and last
processor in the rank, e.g. in above example the directories would be

    processors6_0-2
    processors6_3-5

and each of the collated files in these contains data of the local ranks
only. The same naming also applies when e.g. running decomposePar:

decomposePar -fileHandler collated -ioRanks '(0 3)'

New: Distributed data
---------------------

The individual root directories can be placed on different hosts with different
paths if necessary.  In the current framework it is necessary to specify the
root per slave process but this has been simplified with the option of specifying
the root per host with the -hostRoots command line option:

    mpirun -np 6 simpleFoam -parallel -ioRanks '(0 3)' \
        -hostRoots '("machineA" "/tmp/" "machineB" "/tmp")'

The hostRoots option is followed by a list of machine name + root directory, the
machine name can contain regular expressions.

New: hostCollated
-----------------

The new hostCollated fileHandler automatically sets the 'ioRanks' according to
the host name with the lowest rank e.g. to run simpleFoam on 6 processors with
ranks 0-2 on machineA and ranks 3-5 on machineB with the machines specified in
the hostfile:

    mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler hostCollated

This is equivalent to

    mpirun -np 6 --hostfile hostfile simpleFoam -parallel -fileHandler collated -ioRanks '(0 3)'

This example will write directories:

    processors6_0-2/
    processors6_3-5/

A typical example would use distributed data e.g. no two nodes, machineA and
machineB, each with three processes:

    decomposePar -fileHandler collated -case cavity

    # Copy case (constant/*, system/*, processors6/) to master:
    rsync -a cavity machineA:/tmp/

    # Create root on slave:
    ssh machineB mkdir -p /tmp/cavity

    # Run
    mpirun --hostfile hostfile icoFoam \
        -case /tmp/cavity -parallel -fileHandler hostCollated \
        -hostRoots '("machineA" "/tmp" "machineB" "/tmp")'

Contributed by Mattijs Janssens
2018-03-21 12:42:22 +00:00

240 lines
6.7 KiB
C++

/*---------------------------------------------------------------------------*\
========= |
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\\ / O peration |
\\ / A nd | Copyright (C) 2017-2018 OpenFOAM Foundation
\\/ M anipulation |
-------------------------------------------------------------------------------
License
This file is part of OpenFOAM.
OpenFOAM is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
You should have received a copy of the GNU General Public License
along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>.
Class
Foam::decomposedBlockData
Description
decomposedBlockData is a List<char> with IO on the master processor only.
SourceFiles
decomposedBlockData.C
\*---------------------------------------------------------------------------*/
#ifndef decomposedBlockData_H
#define decomposedBlockData_H
#include "IOList.H"
#include "regIOobject.H"
#include "UPstream.H"
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
namespace Foam
{
/*---------------------------------------------------------------------------*\
Class decomposedBlockData Declaration
\*---------------------------------------------------------------------------*/
class decomposedBlockData
:
public regIOobject,
public List<char>
{
protected:
// Protected data
//- Type to use for gather
const UPstream::commsTypes commsType_;
//- Communicator for all parallel comms
const label comm_;
// Protected member functions
//- Helper: determine number of processors whose recvSizes fits
// ito maxBufferSize
static label calcNumProcs
(
const label comm,
const off_t maxBufferSize,
const labelUList& recvSizes,
const label startProci
);
//- Read data into *this. ISstream is only valid on master.
static bool readBlocks
(
const label comm,
autoPtr<ISstream>& isPtr,
List<char>& data,
const UPstream::commsTypes commsType
);
public:
TypeName("decomposedBlockData");
// Constructors
//- Construct given an IOobject
decomposedBlockData
(
const label comm,
const IOobject&,
const UPstream::commsTypes = UPstream::commsTypes::scheduled
);
//- Construct given an IOobject and for READ_IF_MODIFIED a List<char>
decomposedBlockData
(
const label comm,
const IOobject&,
const UList<char>&,
const UPstream::commsTypes = UPstream::commsTypes::scheduled
);
//- Construct by transferring the IOList contents
decomposedBlockData
(
const label comm,
const IOobject&,
const Xfer<List<char>>&,
const UPstream::commsTypes = UPstream::commsTypes::scheduled
);
//- Destructor
virtual ~decomposedBlockData();
// Member functions
//- Read object
virtual bool read();
//- Write separated content. Assumes content is the serialised data
// and that the master data contains a header
virtual bool writeData(Ostream&) const;
//- Write using given format, version and compression
virtual bool writeObject
(
IOstream::streamFormat,
IOstream::versionNumber,
IOstream::compressionType,
const bool valid
) const;
// Helpers
//- Read header. Call only on master.
static bool readMasterHeader(IOobject&, Istream&);
//- Helper: write FoamFile IOobject header
static void writeHeader
(
Ostream& os,
const IOstream::versionNumber version,
const IOstream::streamFormat format,
const word& type,
const string& note,
const fileName& location,
const word& name
);
//- Read selected block (non-seeking) + header information
static autoPtr<ISstream> readBlock
(
const label blocki,
Istream& is,
IOobject& headerIO
);
//- Read master header information (into headerIO) and return
// data in stream. Note: isPtr is only valid on master.
static autoPtr<ISstream> readBlocks
(
const label comm,
const fileName& fName,
autoPtr<ISstream>& isPtr,
IOobject& headerIO,
const UPstream::commsTypes commsType
);
//- Helper: gather single label. Note: using native Pstream.
// datas sized with num procs but undefined contents on
// slaves
static void gather
(
const label comm,
const label data,
labelList& datas
);
//- Helper: gather data from (subset of) slaves. Returns
// recvData : received data
// recvOffsets : offset in data. recvOffsets is nProcs+1
static void gatherSlaveData
(
const label comm,
const UList<char>& data,
const labelUList& recvSizes,
const label startProc,
const label nProcs,
List<int>& recvOffsets,
List<char>& recvData
);
//- Write *this. Ostream only valid on master. Returns starts of
// processor blocks
static bool writeBlocks
(
const label comm,
autoPtr<OSstream>& osPtr,
List<std::streamoff>& start,
const UList<char>& masterData,
const labelUList& recvSizes,
// optional slave data (on master)
const PtrList<SubList<char>>& slaveData,
const UPstream::commsTypes,
const bool syncReturnState = true
);
//- Detect number of blocks in a file
static label numBlocks(const fileName&);
};
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
} // End namespace Foam
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
#endif
// ************************************************************************* //