19.2.7. PMIx_Fence

PMIx_Fence, PMIx_Fence_nb — Execute a barrier synchronization across a set of processes, optionally collecting data posted via PMIx_Put.

19.2.7.1. SYNOPSIS

#include <pmix.h>

pmix_status_t PMIx_Fence(const pmix_proc_t procs[], size_t nprocs,
                         const pmix_info_t info[], size_t ninfo);

pmix_status_t PMIx_Fence_nb(const pmix_proc_t procs[], size_t nprocs,
                            const pmix_info_t info[], size_t ninfo,
                            pmix_op_cbfunc_t cbfunc, void *cbdata);

19.2.7.1.1. Python Syntax

from pmix import *

foo = PMIxClient()
# ... after a successful foo.init() ...
# the peers is a list of Python ``pmix_proc_t`` dictionaries
peers = [{'nspace': "testnspace", 'rank': PMIX_RANK_WILDCARD}]
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_COLLECT_DATA,
           'value': True, 'val_type': PMIX_BOOL}]
rc = foo.fence(peers, pydirs)

# the non-blocking form returns as soon as the request has been accepted
# and reports the result by executing a callback on the PMIx progress
# thread. The callback is run if and only if the call returned
# PMIX_SUCCESS, and must not itself make a blocking PMIx call.
def donecb(status, cbdata):
    print("fence completed:", foo.error_string(status))
rc = foo.fence_nb(peers, pydirs, donecb, "mycbdata")

19.2.7.2. INPUT PARAMETERS

procs: Pointer to an array of pmix_proc_t structures naming the processes that are to participate in the barrier. A NULL value indicates that the fence is to span all processes in the caller’s own namespace. A rank of PMIX_RANK_WILDCARD in any element indicates that all processes in that namespace are participating.
nprocs: Number of elements in the procs array.
info: Pointer to an array of pmix_info_t(5) structures conveying directives that qualify the operation (see DIRECTIVES). A NULL value is supported when no directives are desired.
ninfo: Number of elements in the info array.

The non-blocking form takes two additional parameters:

cbfunc: Callback function of type pmix_op_cbfunc_t to be invoked when the fence completes.
cbdata: Opaque pointer that is passed, unmodified, to cbfunc.

19.2.7.3. DESCRIPTION

Execute a barrier across the specified processes. PMIx_Fence is the blocking form: it does not return until all participating processes have entered the fence (or the operation fails). PMIx_Fence_nb is the non-blocking form: it returns immediately, and the provided cbfunc is invoked with the final status once the operation completes.

Passing NULL for procs indicates that the fence is to span all processes in the caller’s namespace, equivalent to passing a single pmix_proc_t containing the caller’s namespace and a rank of PMIX_RANK_WILDCARD. The ordering of entries in procs has no significance. However, all processes engaged in a given fence operation must use the same method to identify the participants: callers that describe the target set using PMIX_RANK_WILDCARD are not matched with callers that list the individual processes of a namespace explicitly. A group identifier appearing in procs is first expanded into that group’s member processes.

The calling process must itself be among the participants; if it is not, the operation returns PMIX_ERR_NOT_A_MEMBER.

By default, and for scalability reasons, PMIx_Fence performs only synchronization and does not return the data posted by participants. Setting the PMIX_COLLECT_DATA directive causes the barrier to additionally collect all committed PMIx_Put(3) data from the participants, making it locally available to each participant at the end of the operation. Collected data is cached at the server to reduce memory footprint and is retrieved as needed via PMIx_Get(3).

PMIx_Fence and PMIx_Fence_nb are collective operations. The PMIx server library aggregates the participation of its local clients, passing a single request to the host environment once all local participants have called the API; the host then executes the collective across all participating nodes.

As with all non-blocking PMIx APIs, callers of PMIx_Fence_nb must keep the procs and info arrays valid until cbfunc is invoked.

19.2.7.4. DIRECTIVES

The following attributes are relevant to this operation. The first two are required to be supported by all PMIx libraries; the remainder are optional and depend on the implementation and host environment.

PMIX_COLLECT_DATA (bool) — collect all data posted by the participants via PMIx_Put(3) and committed via PMIx_Commit, making the collection locally available to each participant at the end of the operation. By default this also includes job-level information locally generated by the PMIx servers, unless excluded via PMIX_COLLECT_GENERATED_JOB_INFO. The default behavior of the fence is not to collect data.
PMIX_COLLECT_GENERATED_JOB_INFO (bool) — collect all job-level information (reserved keys) that was locally generated by the PMIx servers.
PMIX_ALL_CLONES_PARTICIPATE (bool) — all clones of the calling process must participate in the collective operation.
PMIX_TIMEOUT (int) — maximum time, in seconds, for the fence to execute before the host declares an error. This helps avoid “hangs” caused by a programming error that prevents one or more processes from reaching the fence.

Note

The PMIX_COLLECTIVE_ALGO and PMIX_COLLECTIVE_ALGO_REQD attributes, used in earlier releases to request specific collective algorithms, are deprecated and should not be used in new code.

19.2.7.5. RETURN VALUE

For the blocking form, PMIX_SUCCESS indicates that the barrier completed successfully and any collected data is available for retrieval. For the non-blocking form, a return of PMIX_SUCCESS indicates only that the request was accepted for processing and the final status will be delivered to cbfunc.

PMIX_SUCCESS — the fence completed successfully.
PMIX_OPERATION_SUCCEEDED — (non-blocking form) the request was satisfied immediately — for example, when the collective involved only processes on the local node — and cbfunc will not be called.
PMIX_ERR_NOT_A_MEMBER — the calling process is not among the named participants.
PMIX_ERR_BAD_PARAM — an invalid argument was supplied (e.g., a NULL procs array with a non-zero nprocs).
PMIX_ERR_UNREACH — the local PMIx server could not be reached.
PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.
PMIX_ERR_INIT — the PMIx library has not been initialized.
PMIX_ERR_WOULD_BLOCK — the call would have blocked the PMIx progress thread from within that thread. See PROGRESS THREAD RESTRICTION.

A NULL cbfunc passed to PMIx_Fence_nb makes that call blocking: it does not return until the fence completes, and its return value is the result of the fence rather than an indication that the request was accepted. That is the general PMIx convention for a non-blocking entry point handed no callback. It can only be honored where the operation’s entire result is a status, as it is here — an entry point that has no way to hand back what the caller asked for, such as PMIx_Get_nb(3), rejects a NULL cbfunc with PMIX_ERR_BAD_PARAM instead.

For a singleton process there are no peers to synchronize with, so the blocking form returns PMIX_SUCCESS (and the non-blocking form PMIX_OPERATION_SUCCEEDED) immediately without contacting a server.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

19.2.7.6. PROGRESS THREAD RESTRICTION

A blocking PMIx call must not be made from within the PMIx progress thread. Any code the library itself invokes runs on that thread: an event handler registered through PMIx_Register_event_handler(3), a callback passed to a non-blocking PMIx API, and — in a server or tool — the completion of a host-module up-call. A blocking call waits for work that the progress thread has to perform, so making one from that thread waits for itself and never returns. The PMIx Standard disallows it, and there is no way for an implementation to service such a request.

Where this call has a blocking form — including the blocking behavior a non-blocking entry point adopts when it is passed a NULL cbfunc — that form detects the situation and returns PMIX_ERR_WOULD_BLOCK immediately, accompanied by a diagnostic naming the call. Nothing is done and no callback is invoked.

PMIX_ERR_WOULD_BLOCK here is not a transient condition to retry: it reports a call that cannot be serviced from where it was made. Reissue it as the non-blocking form with a callback, or from a thread of your own.