6.2. Client Init/Finalize Lifecycle and State Management

This document specifies how PMIx manages the state associated with a client across the PMIx_Init / PMIx_Finalize boundary. It has two halves: the server-side management of the per-client and per-namespace state the server holds on a client’s behalf, and the client-side teardown of the client library itself. Its purpose is to define the intended behavior precisely enough to implement and test it, and in particular to guarantee one property:

The cycling guarantee

A client may cycle PMIx_Init → (work: fences, gets, puts, events, …) → PMIx_Finalize and then PMIx_Init again, any number of times, reusing the same namespace and rank, and each new PMIx_Init must present a fresh interaction state on the server. No state produced in one cycle may leak into the next. The host environment is not required to deregister and re-register the client between cycles for this to hold.

For a code-oriented view of the socket connection that underlies each init, see Transport Layer (Client/Server/Tool Connections). This document is concerned with the state lifecycle, not the transport.

6.2.1. The Problem

When a client calls PMIx_Init, the server accumulates state on its behalf: a peer object holding the connection, event-loop registrations, tag counters, and a datastore (gds) module assignment; entries in collective trackers as the client fences; event and I/O-forwarding registrations; cached notifications; and committed key/value data. The server also carries longer-lived, job-scoped state in the namespace object: the roster of ranks, the count of local processes, the GDS job data, and network/programming-model resources reserved for the job.

A single PMIx_Init/PMIx_Finalize pass, followed by the host tearing the job down, is the common case and works. The difficulty is the cycling case. PMIx_Finalize on the client side tears the client library all the way down (including the class system), so a subsequent PMIx_Init is a brand-new library instance that opens a new socket and reconnects. On the server, the namespace and the rank are still registered — the host did not tear them down — so the second init must reconnect onto existing job-scoped state while starting from a clean per-client slate. If per-client state from the first cycle survives into the second, the client sees stale data, duplicated accounting, or leaked objects.

The design below draws a sharp line between job-scoped state, which belongs to the namespace and persists until the host deregisters it, and client-scoped state, which belongs to a single init/finalize cycle and must not survive one.

6.2.2. The Actors

pmix_namespace_t (nptr): The job. Holds the ranks list, nlocalprocs (the number of local clients registered for this namespace), nfinalized (how many of those have finalized), the compatibility modules negotiated for the job, the namespace-level epilog, and the association to the GDS job data. Persists until the host calls PMIx_server_deregister_nspace.
pmix_rank_info_t (info): One entry per rank on nptr->ranks. Records the rank’s identity (pname), uid/gid, the host’s server_object, the index (peerid) of the peer object serving this rank in the server’s clients array, and proc_cnt — the number of live clones of this rank (see below). This is registration state: it is created by PMIx_server_register_client (or lazily on first connect) and lives until the client or the namespace is deregistered. It is not torn down by finalize.
pmix_peer_t (peer): The per-connection object: socket, send/receive queues, event registrations, dynamic-tag counters, the finalized flag, the peer’s assigned gds module, and the peer-level epilog. Held in pmix_server_globals.clients and pointed to by info->peerid. This is the primary carrier of client-scoped state.
Clones: A client may fork/exec a child that also calls PMIx_Init using the same namespace and rank as its parent — in PMIx terminology, a clone of that client. Each clone opens its own socket and is tracked by info->proc_cnt. A rank is only fully finalized when the original client and all of its clones have finalized — i.e. when proc_cnt returns to zero.

6.2.3. The Three Teardown Triggers

Server-side cleanup is driven by exactly three events. Each has a distinct, non-overlapping responsibility. The remainder of this document is normative: “must” denotes a required behavior.

Client finalize (all clones included)
A client, together with all of its clones, has finalized — the rank’s proc_cnt has dropped to zero. The peer object serving that rank is left in place as an inert tombstone at its existing client-array slot (it is not freed, its slot is not nulled, and info->peerid is not moved), while the rank’s registration (the pmix_rank_info_t on nptr->ranks) is preserved so the client can init again. The tombstone is reclaimed later, at the next reconnect for the rank or at namespace deregistration. See Tombstoning the peer on finalize. A tool peer takes a lighter path — it is freed at socket-close rather than tombstoned, because nothing resolves a tool through info->peerid; see Tombstone mechanics.
Host deregisters a client (PMIx_server_deregister_client)
The server removes the peer object from the namespace and releases it. See Deregistering a client.
Host deregisters the namespace (PMIx_server_deregister_nspace)
The server removes all GDS-registered job data for the namespace, removes and releases all peer objects belonging to the namespace, and cleans up all remaining references to the namespace. See Deregistering the namespace.

The essential distinction is that finalize is reversible and deregistration is terminal. Finalize leaves the registration intact so the client can init again; only the host, by deregistering, declares that the client or the whole job is gone for good and its objects may be freed.

6.2.4. Tombstoning the Peer on Finalize

This section describes the client peer path, where the tombstone is required; a tool peer is retired more simply (freed at socket-close) and is covered under Tombstone mechanics below.

When the last live process for a client rank finalizes (proc_cnt reaches zero), the server must not free the peer object, null its client-array slot, or move info->peerid at socket-close time. Instead it leaves the peer in place as an inert tombstone:

Leave the object at its existing slot in pmix_server_globals.clients with finalized == true. By the time pmix_server_peer_finalized runs, lost_connection has already stopped the peer’s send/receive events and closed its socket, and the finalized flag short-circuits every send path (the PMIX_PTL_SEND_* macros and PMIX_SERVER_QUEUE_REPLY). The tombstone is therefore harmless: it cannot arm a doomed send on its closed socket, yet it keeps info->peerid pointing at a real, resolvable peer object for the rank.
Leave the rank’s pmix_rank_info_t on nptr->ranks (the registration), leave the namespace’s GDS job data in place, and leave info->peerid unchanged.
Reduce only the rank’s live-process count (proc_cnt). The finalized count (nfinalized) is left as-is — the tombstone is still a finalized peer and must keep being counted as one.

The tombstone is reclaimed at a later, quiescent point:

On reconnect. When the client calls PMIx_Init again, the connection handler finds the stale tombstone at info->peerid, frees it (nulls the slot, drops the nfinalized count it was holding, and PMIX_RELEASEs it), then allocates a fresh peer for the rank. The prior cycle has fully finalized by the time a new init arrives, so no collective from it can still be in flight — this is a safe point to mutate the array and the count.
On deregistration. If the rank never reconnects (the common case for a spawned child that runs once and exits), the tombstone is freed when the host deregisters the namespace.

Why leave a tombstone rather than free the peer at socket-close? Two earlier designs freed (or destruct/reconstructed) the object the moment the socket dropped, from inside lost_connection. Both mutated shared per-namespace/per-rank state — nulling the clients slot, and in particular clearing info->peerid to -1 — while a different process’s spawn/connect/disconnect collective or direct-modex get was still walking the clients array and resolving ranks through info->peerid. A rank whose peerid had just been cleared could no longer be resolved to its GDS module, so the in-flight collective or get stalled: an architecture- and timing-sensitive hang of multi-local-process MPI_Comm_spawn. Leaving the finalized peer in place keeps every peerid valid and every array slot populated, so nothing a concurrent collective touches changes underneath it; the object is reclaimed only once no such operation can be in flight.

A tombstone whose slot has been taken over by a newer peer for the same rank — a fork/exec’d clone that is still running, or a fast re-init that reconnected before the socket-close was processed — is no longer named by info->peerid. Such a stranded object is freed immediately in pmix_server_peer_finalized (nulling only its own slot, never the live referenced one, and never touching info->peerid), so a departing clone or a raced reconnect cannot leak a peer or its finalized count.

Job-scoped accounting must stay honest across cycles. The rank’s proc_cnt tracks live processes and returns to zero when the last one departs; nfinalized counts finalized peers and is decremented only when a tombstone is actually reclaimed (on reconnect) or freed (stranded), so that nlocalprocs == nfinalized remains an accurate “all local processes finalized” test no matter how many times clients cycle.

What finalize does not do: Finalize does not remove the rank from nptr->ranks, does not drop the namespace’s GDS job data, and does not touch network or programming-model resources reserved for the job. Those are job-scoped and survive until the host deregisters. Transient, client-scoped registrations that are keyed to the peer — event registrations, I/O-forwarding requests, in-flight direct-modex requests, and cached notifications targeting the client — are pruned as part of the teardown so they do not linger.

6.2.4.1. Tombstone mechanics

The tombstone is driven from lost_connection in the transport layer when a cleanly-finalized client’s socket drops, and its correctness rests on two per-namespace counters staying honest across every cycle.

Peer states. A client peer moves through three states:

State	`finalized`	`sd`	in `proc_cnt`?
Active	`false`	≥ 0	yes
Finalizing (`FINALIZE_CMD` seen, socket not yet dropped)	`true`	≥ 0	yes
Tombstone (socket dropped, kept at `info->peerid`)	`true`	−1	no

A tombstone is deliberately left with finalized == true so it stays inert to every finalized-guarded send path while it waits at info->peerid to be reclaimed. It is removed from proc_cnt (its process is gone) but remains counted in nfinalized (it is still a finalized peer), and it stays resolvable through info->peerid so a concurrent collective or get can still find the rank’s GDS module.

The two counters.

info->proc_cnt is the number of live processes (clones) for the rank. It is incremented in the connection handler when a peer connects and decremented on every peer departure — a clean finalize drop and an abnormal termination. Decrementing on abnormal death too (in lost_connection) keeps a later clone’s tombstone/reclaim accounting correct.
nptr->nfinalized is the number of client peers currently in the finalized == true state, including tombstones. It is incremented when FINALIZE_CMD is received and decremented only when a tombstone is reclaimed — on reconnect, or when a stranded tombstone is freed. Keeping this balanced is what lets the nlocalprocs == nfinalized test at namespace teardown fire the programming-model “all local processes finalized” callback. Because a tombstone stays counted in nfinalized until it is reclaimed, that equality is already satisfied before namespace teardown begins and holds for every rank the teardown then walks. The callback must therefore be fired only once per namespace — a local_app_fini_fired latch on the namespace guards it — rather than once per rank as the ranks are removed.

Finalize on socket drop. lost_connection in the transport layer routes a cleanly-finalized peer whose socket has now dropped to pmix_server_peer_finalized — a pure client so it can be tombstoned, and a pure tool so it can be freed (a peer that is both a client and a tool is left for its tool reconnect/deregistration reclaim; see below). pmix_server_peer_finalized decrements proc_cnt, then, if the peer is still the rank’s referenced peer (info->peerid == peer->index), retires it according to its role:

An application client (PMIX_PEER_IS_CLIENT) is left in place as a tombstone and nothing else is touched — a concurrent collective or get belonging to another local process may still resolve this rank through info->peerid (see below), so its shared state must not change here.
A pure tool is instead freed at once, and info->peerid is repointed — to a surviving tool sibling sharing the same info if one is still live, else to -1. A tool is not an application-job member: it has its own namespace and is never the peer a job collective or direct-modex get resolves through info->peerid, so the tombstone that protects that race is unnecessary. And because each tool init is assigned a fresh namespace, a finalized tool never reconnects onto the same rank to be reclaimed there — leaving it would leak one peer per init/finalize cycle. Freeing it also restores the post-finalize info->peerid == -1 state that process_tool_request relies on to recognize a reconnecting tool.

Otherwise a newer peer already owns the rank’s slot and this object is stranded, so it is freed at once regardless of role: its own clients slot is nulled (never the live referenced one), nfinalized is decremented, and it is PMIX_RELEASEd. A reference still held by a pending non-blocking collective (whose caddy sits on the tracker’s local_cbs) or an active sensor is harmless in every case: a tombstone is simply retained a while longer, and a released object is not torn down until that last holder drops its reference.

Reclaim on reconnect. Before allocating a fresh peer for a reconnecting rank, the connection path reclaims any tombstone the rank left behind. For a client this is the main connection handler, which checks info->peerid: if it names a finalized peer and the rank has no live process (proc_cnt == 0), that tombstone is reclaimed — its slot nulled, nfinalized decremented, and the object released — and only then is a new peer allocated. A tool that was registered as a client (a launcher child) keeps the client tombstone semantics above but reconnects through process_tool_request rather than the client handler; that path performs the same reclaim on the finalized peer at info->peerid before assigning the reconnecting tool its fresh slot. Either way info->peerid is then repointed, by the normal connection assignment, to the freshly allocated peer.

``peerid`` stays valid for clients. Because a client finalize never clears info->peerid to -1, a local PMIx_Get for the rank (which resolves committed data only while info->peerid names a resolvable peer) continues to work against the client tombstone until the rank either reconnects or is deregistered. A pure tool, freed at socket-close, instead leaves info->peerid == -1 — which is correct, because no other process resolves a tool’s rank this way.

6.2.5. Deregistering a Client

PMIx_server_deregister_client is the host’s declaration that a specific rank is gone and will not init again. Unlike finalize — which tombstones the peer object but keeps the rank’s registration so it can init again — deregistration also removes the registration:

Remove the peer object from the namespace: release the reference the namespace/registration holds, null out the client-array slot identified by info->peerid, and remove the pmix_rank_info_t from nptr->ranks.
Release the peer object.
Restore any job resources that were charged to the rank (network allocations, monitoring/sensors) and honor the peer’s epilog, exactly as when a client terminates.

After a client is deregistered, the rank no longer exists on the server; a subsequent connection for that rank would be rejected unless the host re-registers it.

6.2.6. Deregistering the Namespace

PMIx_server_deregister_nspace is the host’s declaration that the entire job is gone. This is the terminal, whole-namespace teardown and must reclaim everything the namespace owned:

Remove all GDS-registered job data for the namespace (gds del of the namespace), so no key/value or job-info data survives.
Remove and release all peer objects still belonging to the namespace (any rank whose processes have not all finalized), nulling their client-array slots and freeing every pmix_rank_info_t on nptr->ranks.
Release the network and programming-model resources reserved for the job, run the namespace epilog, and prune every remaining reference to the namespace: event and I/O-forwarding registrations, cached notifications targeting any of its procs, and finally the namespace object itself.

Nothing that names the namespace may remain after this call returns.

6.2.7. State Ownership Summary

The following table classifies the state and states which trigger frees it. “Released” means the object is freed and its client-array slot nulled; “Tombstoned” means it is left inert in place and reclaimed later; “Kept” means the trigger leaves it in place.

State	Client finalize (all clones)	Deregister client	Deregister nspace
Peer object (`pmix_peer_t`)	Tombstoned (kept at slot; reclaimed on reconnect/deregister)	Released, slot nulled	Released, slot nulled
Event / IOF / direct-modex regs, cached notifications for the peer	Pruned	Pruned	Pruned
Peer epilog	Executed	Executed	Executed (peer) + namespace epilog
Rank info (`pmix_rank_info_t` on `nptr->ranks`)	Kept (registration preserved)	Removed, released	Removed, released
GDS job data for the namespace	Kept	Kept	Removed
Network / programming-model job resources	Kept	Rank’s share restored	Released (whole job)
Namespace object (`pmix_namespace_t`)	Kept	Kept	Released; all references purged

6.2.8. Client-Side Teardown

Everything above concerns the state a server holds for a client. The cycling guarantee has a second half: the client’s own library must tear itself down on PMIx_Finalize so that a subsequent PMIx_Init in the same process starts from a clean slate. Unlike the server side — where finalize deliberately preserves registration state — the client side has no such asymmetry to manage: a client PMIx_Finalize (once the init reference count reaches zero) is a full teardown of the library, and a later PMIx_Init is a brand-new library instance that opens a fresh socket and reconnects.

The teardown runs PMIx_Finalize → pmix_rte_finalize → framework closes plus pmix_finalize_util and the release of the progress thread; it is the mirror image of PMIx_Init → pmix_rte_init → pmix_init_util plus the framework opens. The governing principle is symmetry: finalize must undo exactly what init did, leaving no residue that the next init would either trip over or skip.

The reference-count guard: PMIx_Init is reference-counted: nested init/init/…/finalize sequences only tear down on the last finalize, when the counter returns to zero. The full teardown described here applies to that final finalize. A nested (non-final) finalize must leave the library running and untouched.

Three requirements make the symmetry real, and an implementation must uphold all three:

Every one-time-init latch must be reset. Any static boolean or counter that guards a “do this once” step in init (module registration, key creation, subsystem setup) must be cleared by the corresponding finalize, so the step runs again on the next init. Leaving a latch set silently skips re-initialization on the second cycle — the failure is not a crash at finalize but wrong or missing state later. The multi-select framework initialized flags, the MCA parameter-registration latch, and the attribute/output/show-help subsystems all follow this rule and are the model to copy.
Every freed global must be nulled, and every per-cycle flag reset. A global pointer whose target is freed during finalize must be set to NULL in the same step. Two hazards follow from not doing so:
- Dangling reuse. A non-NULL pointer to freed memory defeats the if (NULL == p) guards that would otherwise make a between-cycles access safe; it reads as “still valid” and yields a use-after-free. The event base, the shared progress-thread tracker, and the cached server peer are pointers of this kind.
- Stale mode flags. A flag that records how this cycle was set up — most importantly the singleton indicator, which records that the process came up with no server and therefore constructed the server-side I/O-forwarding lists — must be reset on every fresh init, not only when it happens to be toggled. A stale singleton flag carried from a no-server cycle into a with-server cycle drives finalize to tear down lists the with-server path never built.
Init and finalize of the utility layer must stay in lockstep. pmix_init_util opens a fixed set of low-level subsystems (output, install-dirs, show-help, the keyval parser, the MCA base and its variable system, the network and interface helpers). Each must be closed on finalize. Whether the close lives in pmix_finalize_util or inline in pmix_rte_finalize is an implementation choice, but the two lists must be kept in correspondence: a subsystem added to init without a matching teardown persists across a cycle and breaks the clean-slate guarantee for the next init.

Platform caveat: destructor support: On platforms that provide neither a compiler destructor attribute nor linker -fini support, the library cannot guarantee cleanup at unload, and re-initialization after finalize is explicitly unsupported — a second PMIx_Init returns PMIX_ERR_NOT_SUPPORTED rather than risk an inconsistent restart. The cycling guarantee therefore applies only where destructor or -fini support is present (the mainstream case).

6.2.9. Tool-Side Teardown

The tool role has the same cycling guarantee as the client: a process must be able to cycle PMIx_tool_init → work → PMIx_tool_finalize repeatedly, each init a clean slate. PMIx_tool_init / PMIx_tool_finalize are a separate entry-point pair from the client PMIx_Init / PMIx_Finalize — they do not share the client’s finalize code — so the three symmetry requirements above have to be satisfied independently in the tool path. Two of them bite the tool specifically:

The reference-count guard and the one-time latch

PMIx_tool_init is reference-counted through its own counter, and it sets the shared one-time-init latch (pmix_globals.init_called). PMIx_tool_finalize must decrement that counter, tear down only on the last matching finalize, and reset the latch. Forgetting the reset is the most visible cycling failure: the second PMIx_tool_init sees the latch still set, concludes the library is already initialized, and returns PMIX_ERR_INIT — the tool never comes back up.

The tool constructs server state every init

Because a tool may itself act as a server (a launcher connects up to its own server while listening for its children), PMIx_tool_init unconditionally initializes the server_globals bookkeeping — constructing the client array and every server-side list — on every init, whether or not the tool will act as a server. Tool finalize must therefore destruct all of those lists, not just the subset a particular run happened to populate; any list left un-destructed leaks its contents each cycle when the next init reconstructs over it. For the same reason the tool must free (and NULL) the tmpdir and system_tmpdir strings: the init-time code refreshes them only when they are NULL, so a surviving pointer both leaks and silently pins the previous cycle’s temporary-directory choice, ignoring a changed environment on the next init. The tool must likewise destruct the client-side peers array backing store and the static IOF sinks, exactly as the client finalize does.

The reset of the server module latch (pmix_server_globals.module_set, set by PMIx_tool_set_server_module) is not part of this pair: that latch is owned by a separate registration call a launcher makes outside PMIx_tool_init, and the server reference finalizer does not clear it either. A launcher that re-registers its server module between cycles is therefore out of scope here and handled separately.

6.2.10. Invariants

An implementation of the above must uphold these invariants:

No cross-cycle carryover. The finalized peer serving a rank is reclaimed (freed) no later than the rank’s next reconnect, at which point a fresh object is allocated, so a new cycle cannot inherit queues, event registrations, tag counters, or datastore assignments from the previous one.
No orphaned peers. Cycling init/finalize must not accumulate unreferenced peer objects in the clients array. A finalized client peer is left as a single tombstone at info->peerid and is reclaimed when the rank reconnects (or when the namespace is deregistered); a peer that a newer connection has already displaced, and a finalized tool peer, are freed immediately. Either way at most one finalized object per rank is ever kept, and only while the rank’s registration itself persists — including across repeated tool init/finalize cycles that reuse the same namespace and rank, whose reconnect reclaim runs in process_tool_request.
No shared-state mutation while a collective is in flight. Finalize must not free a peer, null a live clients slot, or move a live info->peerid from lost_connection: a concurrent spawn/connect/disconnect collective or direct-modex get may be walking that state for another process. All such mutation is deferred to a quiescent point (reconnect or deregistration), or confined to a stranded object no live peerid names.
No counter drift. Per-namespace finalized accounting and per-rank proc_cnt must return to values consistent with the actual number of connected processes after each cycle, so that the job-level “all local processes finalized” condition and any resource-release keyed on it remain correct no matter how many times clients cycle.
Registration outlives finalize; only deregistration is terminal. The pmix_rank_info_t and the client-array slot persist across finalize and are removed only by an explicit host deregister. GDS job data and job resources persist until PMIx_server_deregister_nspace.
The client library reinitializes from a clean slate. After the final PMIx_Finalize, no static init-once latch remains set, no global pointer references freed memory, and no per-cycle mode flag retains a value from the previous cycle. The next PMIx_Init behaves exactly as the first.