19.2.13. PMIx_Connect

PMIx_Connect, PMIx_Connect_nb — Record a set of processes as “connected” so that the host environment treats them as a single assemblage for fault and event-notification purposes.

19.2.13.1. SYNOPSIS

#include <pmix.h>

pmix_status_t PMIx_Connect(const pmix_proc_t procs[], size_t nprocs,
                           const pmix_info_t info[], size_t ninfo);

pmix_status_t PMIx_Connect_nb(const pmix_proc_t procs[], size_t nprocs,
                              const pmix_info_t info[], size_t ninfo,
                              pmix_op_cbfunc_t cbfunc, void *cbdata);

19.2.13.1.1. Python Syntax

from pmix import *

foo = PMIxClient()
# ... after a successful foo.init() ...
# the peers is a list of Python ``pmix_proc_t`` dictionaries naming all
# participants (including the caller)
peers = [{'nspace': "testnspace", 'rank': PMIX_RANK_WILDCARD},
         {'nspace': "othernspace", 'rank': PMIX_RANK_WILDCARD}]
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_TIMEOUT,
           'value': 10, 'val_type': PMIX_INT}]
rc = foo.connect(peers, pydirs)

# the non-blocking form returns as soon as the request has been accepted
# and reports the result by executing a callback on the PMIx progress
# thread. The callback is run if and only if the call returned
# PMIX_SUCCESS, and must not itself make a blocking PMIx call.
def donecb(status, cbdata):
    print("connect completed:", foo.error_string(status))
rc = foo.connect_nb(peers, pydirs, donecb, "mycbdata")

19.2.13.2. INPUT PARAMETERS

procs: Pointer to an array of pmix_proc_t structures naming all of the processes that are to be connected. A rank of PMIX_RANK_WILDCARD in any element indicates that all processes in that namespace are participating. Unlike the collective APIs that default to the caller’s own namespace, procs must be non-NULL and nprocs must be greater than zero; otherwise the call returns PMIX_ERR_BAD_PARAM.
nprocs: Number of elements in the procs array.
info: Pointer to an array of pmix_info_t(5) structures conveying directives that qualify the operation (see DIRECTIVES). A NULL value is supported when no directives are desired.
ninfo: Number of elements in the info array.

The non-blocking form takes two additional parameters:

cbfunc: Callback function of type pmix_op_cbfunc_t to be invoked when the connect operation completes.
cbdata: Opaque pointer that is passed, unmodified, to cbfunc.

19.2.13.3. DESCRIPTION

Record the processes specified by the procs array as connected. PMIx_Connect is the blocking form: it does not return until all processes identified in procs have called either PMIx_Connect or PMIx_Connect_nb and the host environment has completed any supporting operations required to meet the terms of the PMIx definition of connected processes (or the operation fails). PMIx_Connect_nb is the non-blocking form: it returns immediately, and the provided cbfunc is invoked with the final status once the operation completes.

The PMIx definition of connected solely implies that the host environment should treat the failure of any process in the assemblage as a reportable event, taking action on the assemblage as if it were a single application. For example, if the environment defaults (in the absence of any application directives) to terminating an application upon failure of any process in that application, then it should terminate all processes in the connected assemblage upon failure of any member. The host environment may choose to assign a new namespace to the connected assemblage and/or assign new ranks to its members for its own internal tracking purposes; whether such namespaces are exposed to clients is implementation defined.

Every element of procs must name a participant, and the calling process must itself be among them; if it is not, the operation returns PMIX_ERR_NOT_A_MEMBER. The ordering of the entries in procs has no significance. However, all processes engaged in a given connect operation must use the same method to identify the participants: callers that describe the target set using PMIX_RANK_WILDCARD are not matched with callers that list the individual processes of a namespace explicitly. A group identifier appearing in procs is first expanded into that group’s member processes.

Upon completion, the server returns any job-level information that participating processes do not already possess. When the connect request spans processes from different namespaces, each participant receives the job-level information for the namespaces other than its own, so that subsequent PMIx_Get(3) queries against the connected peers can be satisfied locally.

Each participant also contributes its own endpoint data to the operation, and what that consists of is worth being precise about: it is exactly the values that process posted with PMIx_Put(3) at PMIX_REMOTE or PMIX_GLOBAL scope, less any reserved key. Because every participant must call PMIx_Connect — that is what makes it a participant — each supplies its own, and no server or host is required to sweep the participants collecting it. The corollary is that an endpoint exchange contains what its participants put and nothing more; information the runtime computed rather than the application posting it, such as fabric endpoint assignments, is job-level data and reaches participants through the job-level information described above.

A process can only engage in one connect operation involving the identical procs array at a time. However, a process can be simultaneously engaged in multiple connect operations, each involving a different procs array.

PMIx_Connect and PMIx_Connect_nb are collective operations. The PMIx server library aggregates the participation of its local clients, passing a single request to the host environment once all local participants have called the API; the host then executes the collective across all participating nodes.

As with all non-blocking PMIx APIs, callers of PMIx_Connect_nb must keep the procs and info arrays valid until cbfunc is invoked.

19.2.13.4. DIRECTIVES

PMIx libraries are not required to directly support any attributes for this operation, but any provided attributes are passed to the host environment for processing. The following attributes are optional and depend on the implementation and host environment:

PMIX_ALL_CLONES_PARTICIPATE (bool) — all clones of the calling process must participate in the collective operation.
PMIX_TIMEOUT (int) — maximum time, in seconds, for the connect to complete before the host declares an error. This helps avoid “hangs” caused by a programming error that prevents one or more processes from reaching the connect.

19.2.13.5. RETURN VALUE

For the blocking form, PMIX_SUCCESS indicates that all participants have connected and any returned job-level data is available. For the non-blocking form, a return of PMIX_SUCCESS indicates only that the request was accepted for processing and the final status will be delivered to cbfunc.

PMIX_SUCCESS — the connect operation completed successfully.
PMIX_ERR_NOT_A_MEMBER — the calling process is not among the named participants.
PMIX_ERR_BAD_PARAM — an invalid argument was supplied (e.g., a NULL procs array or a zero nprocs).
PMIX_ERR_UNREACH — the local PMIx server could not be reached.
PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.
PMIX_ERR_INIT — the PMIx library has not been initialized.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

19.2.13.6. NOTES

Attempting to connect processes solely within the same namespace is essentially a no-op operation. While not explicitly prohibited, a PMIx implementation or host environment may return an error in such cases.

Once processes are connected, the host environment is required to generate a PMIX_ERR_PROC_TERM_WO_SYNC event should any process in the assemblage terminate or call PMIx_Finalize(3) without first disconnecting from the assemblage via PMIx_Disconnect(3). If a job is terminated as a result of that action, the host environment is also required to generate PMIX_ERR_JOB_TERM_WO_SYNC for all jobs that were terminated as a result.

By default, the processes created by PMIx_Spawn(3) are connected to the parent process upon successful launch, following the same semantics described here.

19.2.13.7. PROGRESS THREAD RESTRICTION

A blocking PMIx call must not be made from within the PMIx progress thread. Any code the library itself invokes runs on that thread: an event handler registered through PMIx_Register_event_handler(3), a callback passed to a non-blocking PMIx API, and — in a server or tool — the completion of a host-module up-call. A blocking call waits for work that the progress thread has to perform, so making one from that thread waits for itself and never returns. The PMIx Standard disallows it, and there is no way for an implementation to service such a request.

Where this call has a blocking form — including the blocking behavior a non-blocking entry point adopts when it is passed a NULL cbfunc — that form detects the situation and returns PMIX_ERR_WOULD_BLOCK immediately, accompanied by a diagnostic naming the call. Nothing is done and no callback is invoked.

PMIX_ERR_WOULD_BLOCK here is not a transient condition to retry: it reports a call that cannot be serviced from where it was made. Reissue it as the non-blocking form with a callback, or from a thread of your own.