6.4.1. Overview
PMIx supports two slightly related, but functionally different concepts known as process sets and process groups. This section defines these two concepts and describes how they are utilized, along with their corresponding APIs.
6.4.1.1. Process Sets vs. Groups
A PMIx Process Set is a user-provided or host environment assigned
label associated with a given set of application processes. Processes can
belong to multiple process sets at a time. Users may define a PMIx
process set at time of application execution. For example, if using the
command line parallel launcher prun
, one could specify process sets
as follows:
$ prun -n 4 --pset ocean myoceanapp : -n 3 --pset ice myiceapp
In this example, the processes in the first application will be labeled with a PMIX_PSET_NAME
attribute with a value of ocean while those in the second application will be labeled with an
ice value. During the execution, application processes could lookup the process set attribute
for any process using PMIx_Get
. Alternatively, other executing applications could utilize the
PMIx_Query_info
APIs to obtain the number of declared process sets in the system, a list of
their names, and other information about them. In other words, the process set identifier provides
a label by which an application can derive information about a process and its application - it does
not, however, confer any operational function.
Host environments can create or delete process sets at any time through the
PMIx_server_define_process_set
and
PMIx_server_delete_process_set
APIs. PMIx servers shall
notify all local clients of process set operations via the
PMIX_PROCESS_SET_DEFINE
or PMIX_PROCESS_SET_DELETE
events.
Process sets differ from process groups in several key ways:
Process sets have no implied relationship between their members - i.e., a process in a process set has no concept of a
pset rank
as it would in a process group.Process set identifiers are set by the host environment or by the user at time of application submission for execution - there are no PMIx client API’s provided by which an application can define a process set or change a process set membership. In contrast, PMIx process groups can only be defined dynamically by the application.
Process sets are immutable - members cannot be added or removed once the set has been defined. In contrast, PMIx process groups can dynamically change their membership using the appropriate API’s.
Process groups can be used in calls to PMIx operations. Members of process groups that are involved in an operation are translated by their PMIx library into their native identifier prior to the operation being passed to the host environment. For example, an application can define a process group to consist of ranks 0 and 1 from the host-assigned namespace of 210456, identified by the group id of foo. If the application subsequently calls the PMIx_Fence API with a process identifier of {foo, PMIX_RANK_WILDCARD}, PMIx library will replace that identifier with an array consisting of {210456, 0} and {210456, 1} - the host-assigned identifiers of the participating processes - prior to processing the request.
Process groups can request that the host environment assign a unique size_t context identifier to the group at time of group construction. An MPI library may, for example, use the ID as the MPI communicator identifier for the group.
The two concepts do, however, overlap in that they both involve collections of processes. Users desiring to create a process group based on a process set could, for example, obtain the membership array of the process set and use that as input to PMIx_Group_construct, perhaps including the process set name as the group identifier for clarity. Note that no linkage between the set and group of the same name is implied nor maintained - e.g., changes in process group membership can not be reflected in the process set using the same identifier.
Note
The host environment is responsible for ensuring:
consistent knowledge of process set membership across all involved PMIx servers; and
that process set names do not conflict with system-assigned namespaces within the scope of the set.