.. _group-implementation-label: Group Operations: Theory and Implementation =========================================== .. note:: This is a *developer-level* specification. It describes how the ``PMIx_Group_*`` family is realized inside ``libpmix`` — the client-side and server-side machinery, the two-phase collective model, and the fault-tolerance accounting that keeps a group operation from hanging when a participant is lost. The user-facing semantics are documented in :ref:`Group Construction, Destruction, and Fault Tolerance `; the general server-side collective-tracking model that the group family reuses is specified in :doc:`../collectives/tracking_spec`. Audience: contributors modifying the group code, the collective-tracking service, or a host environment's ``pmix_server_module_t.group`` callback. Two realizations of "group construction" ---------------------------------------- The group family is implemented as **two distinct mechanisms**, chosen per API rather than per group: * **Round-trip collectives.** ``PMIx_Group_construct`` and ``PMIx_Group_destruct`` are true collectives. Each participant's client library packs a command and sends it to its local server with ``PMIX_PTL_SEND_RECV``; the server assembles the local participants, forwards the aggregated operation to the host for global completion, and replies to each local participant. Commands: ``PMIX_GROUP_CONSTRUCT_CMD`` and ``PMIX_GROUP_DESTRUCT_CMD``. * **Event-driven handshakes.** ``PMIx_Group_invite``, ``PMIx_Group_join``, and ``PMIx_Group_leave`` send **no** server command. They are realized entirely through the event-notification subsystem (``PMIx_Notify_event``), with the server's only role being to relay events between its clients and the host. This is why the invite method works without any host collective support, and why ``PMIx_Group_leave`` was re-implemented as an event (its former ``PMIX_GROUP_LEAVE_CMD`` round-trip was never handled by any server switchyard and has been removed). The code lives in two files: * ``src/client/pmix_client_group.c`` — all six client API pairs, the client group tracker, and the client-side fault handling (invite timeout/decline, construct abort, leader-failure watch). * ``src/server/pmix_server_group.c`` — the server block/tracker structures, the local-phase assembly and completion predicate, host handoff, and the server-side fault accounting. Shared membership helpers used by the collectives live in ``src/client/pmix_client_convert.c``. Client side ----------- The group tracker ^^^^^^^^^^^^^^^^^ A single caddy type, ``pmix_group_tracker_t`` (in ``pmix_client_group.c``), carries state across the progress-thread boundary for every API in the family. It is deliberately overloaded — its meaning shifts with the operation — but always carries the standard caddy machinery (``ev``, ``lock``, ``status``, ``cbdata``, cached ``cbfunc``/``opcbfunc``) plus the group identity (``grpid``, ``members``/``nmembers``). For the event-driven paths it additionally holds the per-member bookkeeping used by the invite/join handshake (``responded[]``, ``answered[]``, ``nanswered``, ``optional``), the timeout state (``timer_active``), a one-shot resolution guard (``completed``), and the event-handler registration id (``ref``, seeded to ``SIZE_MAX`` so teardown can tell whether a handler was ever registered). Construct and destruct (round-trip) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``PMIx_Group_construct_nb`` does the real work; the blocking form is a thin wrapper that supplies an internal callback and waits on the caddy lock. The non-blocking form: #. **Expands and validates membership.** When the participants are named explicitly (not the add-members or bootstrap methods), the ``procs`` array is run through ``pmix_client_convert_group_procs()`` so any embedded PMIx *group* identifier is replaced by its member processes, and the caller is checked for inclusion via ``pmix_client_proc_is_included()``. A caller that is not among the expanded participants gets ``PMIX_ERR_NOT_A_MEMBER`` immediately. Both steps are skipped when ``PMIX_GROUP_ADD_MEMBERS`` or ``PMIX_GROUP_BOOTSTRAP`` is present, since those intentionally omit participants. #. **Packs the message** (``construct_msg()``): the command, group ID, participant array, and info array, folding in the caller's own endpoint data (fetched at ``PMIX_REMOTE`` scope and packed as ``PMIX_PROC_DATA``) and prepending the caller's proc ID to any ``PMIX_GROUP_INFO`` array. #. **Dispatches** with ``PMIX_PTL_SEND_RECV(..., construct_cbfunc, cb)`` and returns. ``construct_cbfunc`` unpacks the returned status, final membership, optional context ID, and group info, registers the completed group in ``pmix_client_globals.groups`` (sorted for a consistent cross-participant view), and delivers the results. ``PMIx_Group_destruct_nb`` mirrors this with ``PMIX_GROUP_DESTRUCT_CMD``, sending the membership it holds locally (the server does not store it) and removing the group from the local list on completion. The persistent ``pmix_group_t`` also carries the group's failure policy across the two collectives. ``PMIX_GROUP_NOTIFY_TERMINATION`` is specified at construct time but governs the eventual destruct, and the server keeps no group state between operations — so the blocking construct wrapper captures the flag into the tracker, ``add_group()`` records it in the ``pmix_group_t`` (``notterm``), and ``PMIx_Group_destruct_nb`` re-attaches ``PMIX_GROUP_NOTIFY_TERMINATION`` to the destruct request (unless the caller passed it explicitly, which overrides). The server then reads it off the aggregated destruct block via ``grp_notify_termination()``. ``pmix_client_proc_is_included()`` (in ``pmix_client_convert.c``) is the shared "is the caller covered by this participant array?" predicate — it matches the caller's namespace and accepts ``PMIX_RANK_WILDCARD``, ``PMIX_RANK_LOCAL_NODE``, ``PMIX_RANK_LOCAL_PEERS``, or an exact rank. It was hoisted out of duplicate copies in the fence and connect/disconnect paths so all three collectives share one membership check. Invite / join / leave (event-driven) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The invite method tracks invitees **by identity**. ``invite_setup()`` expands wildcard ranks into concrete members, builds parallel ``responded[]`` (accepters) and ``answered[]`` (accepters *and* decliners *and* terminated) flag arrays, seeds the leader's own slot, and registers a prepended handler for ``PMIX_GROUP_INVITE_ACCEPTED``, ``PMIX_GROUP_INVITE_DECLINED``, and ``PMIX_PROC_TERMINATED``. It then notifies the invitees with ``PMIX_GROUP_INVITED`` over a custom range, and, if a ``PMIX_TIMEOUT`` was supplied, arms a libevent timer. The critical thread-safety rule: the response handler (``invite_handler``) and the timeout handler run on the **progress thread** and must never raise an event themselves (that would deadlock). They only *wake* the operation (``invite_wake``, guarded by the ``completed`` one-shot). All event generation — ``PMIX_GROUP_INVITE_FAILED`` for non-accepters, the ``PMIX_GROUP_CONSTRUCT_COMPLETE`` broadcast, or a ``PMIX_GROUP_CONSTRUCT_ABORT`` — happens afterward on the caller's thread in ``invite_announce()``. A termination event names the departed proc in ``PMIX_EVENT_AFFECTED_PROC`` (not the event source), so ``invite_handler`` attributes it to the right member via that key. ``PMIx_Group_join_nb`` maps the ``pmix_group_opt_t`` to an event code (``PMIX_GROUP_INVITE_ACCEPTED`` / ``PMIX_GROUP_INVITE_DECLINED``) and notifies the leader. On an *accept* with a known leader it also arms the leader-failure watch (below). ``PMIx_Group_leave_nb`` finds the group locally, drops it from the local list immediately, and generates a ``PMIX_GROUP_LEFT`` event ranged to the membership excluding self (naming self in ``PMIX_EVENT_AFFECTED_PROC``). Per its contract it returns once the event is *locally generated*. Remaining members update their local membership when they receive ``PMIX_GROUP_LEFT``, handled in ``pmix_invoke_local_event_hdlr`` (``src/event/pmix_event_notification.c``) alongside the existing group-construct-complete membership handling. The leader-failure watch ^^^^^^^^^^^^^^^^^^^^^^^^^ When a process accepts an invitation, ``setup_leader_watch()`` registers a private, non-blocking, prepended handler keyed on leader termination (``PMIX_PROC_TERMINATED``, ``PMIX_ERR_PROC_ABORTED``, ``PMIX_ERR_PROC_TERM_WO_SYNC``) and on construct resolution (``PMIX_GROUP_CONSTRUCT_COMPLETE`` / ``PMIX_GROUP_CONSTRUCT_ABORT``). If the watched leader (held in ``members[0]``) is lost before the construct resolves, the handler emits ``PMIX_GROUP_LEADER_FAILED`` to this process's own handlers via **non-blocking** ``PMIx_Notify_event`` (the blocking form would deadlock on the progress thread) and tears itself down. It always returns ``PMIX_EVENT_NO_ACTION_TAKEN`` so it never swallows the events the application registered to see. Reselection is entirely application-driven: a process nominates itself by returning ``PMIX_GROUP_LEADER`` in its handler's results, and the outcome is announced with ``PMIX_GROUP_LEADER_SELECTED``. Server side ----------- Block and tracker structures ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Group collectives use a **two-level** tracker (unlike the flat ``pmix_server_trkr_t`` shared by fence/connect/disconnect), because a single group construct can be assembled from several distinct call signatures (leader, followers, bootstrap) and carries group-only state: * ``grp_block_t`` — one per group-ID operation, on the server-global list ``pmix_server_globals.grp_collectives`` (declared in ``src/server/pmix_server_ops.h``). It owns: the operation kind ``grpop`` (a ``pmix_group_operation_t``, initialized to ``PMIX_GROUP_NONE``); the aggregated membership (``pcs``/``npcs``) and directives (``info``/``ninfo``) built by ``aggregate_info``; the list ``mbrs`` of ``grp_trk_t``, **one per participant call** (its size is the number of local contributions); the list ``departed`` of processes lost before contributing; ``nlocal`` (the expected local count) and ``def_complete`` (all participating namespaces registered, so ``nlocal`` is final); ``host_called`` (the local phase is forwarded and frozen); ``need_cxtid``; and the local-phase timeout state (``ev``, ``event_active``). * ``grp_trk_t`` — one per participant call, hung off a block's ``mbrs``. It holds the membership and directives *as passed by that participant*, a back-pointer to its block, and ``local_cbs`` — the list of ``pmix_server_caddy_t`` that both identifies **who actually contributed** (by peer name) and holds the reply destination for each local participant. * ``grp_shifter_t`` — the thread-shift caddy carrying the host's completion result back onto the progress thread. Local-phase assembly and completion ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``pmix_server_group()`` receives a construct/destruct command on the progress thread. It requires a host ``group`` callback (else ``PMIX_ERR_NOT_SUPPORTED``), unpacks the group ID / procs / directives, seeds a default ``PMIX_LOCAL_COLLECTIVE_STATUS = PMIX_SUCCESS`` slot as the last element of the local info array (the slot the fault machinery later overwrites), and locates or creates the block via ``get_tracker()``. A call with ``nprocs == 0`` is a *follower*; a ``PMIX_GROUP_BOOTSTRAP`` call is a *bootstrap* leader — both keep independent trackers (the leader is unknown) and are forwarded to the host immediately, one call per participant, without local aggregation. For the collective (fully-specified) method: * ``check_definition_complete()`` walks all trackers' member procs, verifies every participating namespace is known and fully registered, and — once so — sets ``def_complete`` and the final ``nlocal`` (wildcards contribute the namespace's local process count). Until every namespace is registered the block stays pending, so a client that calls in before its local ``register_client`` event cannot complete the collective early. * ``grp_blk_locally_complete()`` is the **single completion predicate**:: def_complete AND (size(mbrs) + size(departed)) >= nlocal This is the group analog of ``pmix_server_trk_complete()`` used by the fence family. The ``>=`` (rather than ``==``) tolerates fork/exec clones that share a rank and over-count. The local phase is complete once every expected local participant has either **contributed** (a tracker on ``mbrs``) or **departed**. * When the predicate holds and the block has not already been forwarded, ``aggregate_info()`` merges every tracker's procs/info into the block (deduplicating and special-casing ``PMIX_GROUP_ADD_MEMBERS``, ``PMIX_GROUP_BOOTSTRAP``, ``PMIX_PROC_DATA``, and ``PMIX_GROUP_INFO``), the fault decision below is applied, the local-phase timer is deleted, ``host_called`` is set, and the operation is forwarded to ``pmix_host_server.group()``. The host's completion callback ``grpcbfunc`` thread-shifts into ``_grpcbfunc``, which extracts the context ID, final membership, and group info from the host result, stores group info via ``pmix_server_process_grpinfo``, and — for every block matching the ID (bootstrap can create several) — packs each tracker's ``local_cbs`` reply and queues it, then removes and releases the block. Fault tolerance: identity-based accounting ------------------------------------------ The group family reuses the identity-based collective-tracking model specified in :doc:`../collectives/tracking_spec`. The essential idea is that local participation must be tracked **by participant identity**, never by a bare counter, so the lost-connection handler can answer the one question a counter cannot: *"has this specific participant already contributed?"* The completion predicate counts ``contributed ∪ departed`` against ``expected``; the two lost-participant rules are: * **Case A — the participant had already contributed.** Ignore the loss for this collective. The contribution stands, its data stays in the assembled result, and ``nlocal`` is not reduced. The dead peer's queued reply is harmless (its socket is closed). This is the rule that fixes the historical early-completion / data-loss bug. * **Case B — the participant had not yet contributed.** It never will, so record it on ``departed`` (once, deduplicated). Completion is then re-evaluated; if the block is now locally complete it is forwarded exactly as on a normal final contribution. Two entry points reach the same per-block routine ``account_departed()``: * ``pmix_server_grp_peer_lost(peer)`` — called from ``lost_connection()`` in ``src/mca/ptl/base/ptl_base_sendrecv.c`` when a socket drops. A lost connection is not scoped to one group, so it applies ``account_departed`` to **every** block on ``grp_collectives``. (Before this work the lost-connection handler walked only the fence-family ``collectives`` list, so a lost group member hung the construct forever.) * ``pmix_server_grp_member_left(grpid, proc)`` — the deliberate cousin. A ``PMIx_Group_leave`` generates a ``PMIX_GROUP_LEFT`` event that the originating server intercepts in ``src/server/pmix_server_ops.c`` (before loop-detection, so only the originating server acts). It applies ``account_departed`` to the blocks for the **named** group only. ``account_departed()`` skips bootstrap/follower blocks (``nlocal == 0``) and already-forwarded blocks (``host_called``); determines membership by matching ``proc`` against each tracker's ``pcs`` with ``PMIX_CHECK_NAMES`` and prior contribution by matching against each tracker's ``local_cbs`` peer names (Case A vs. Case B); and, if the loss completes the local phase, aggregates, applies the fault decision, and forwards. Callers iterate with a ``FOREACH_SAFE`` macro because the block may be released on error. The abort-vs-survive gate ^^^^^^^^^^^^^^^^^^^^^^^^^^ At each point where a *loss* (or leave) completes a block's local phase, the server decides between aborting and surviving: .. code-block:: c if (0 < pmix_list_get_size(&blk->departed)) { if (PMIX_GROUP_CONSTRUCT == op && !grp_ft_collective(blk)) { abort_construct(blk, PMIX_GROUP_CONSTRUCT_ABORT); return; } if (!(PMIX_GROUP_DESTRUCT == op && grp_notify_termination(blk))) { pmix_server_set_collective_status(blk->info, blk->ninfo, PMIX_ERR_LOST_CONNECTION); } } * ``grp_ft_collective(blk)`` scans the aggregated directives for ``PMIX_GROUP_FT_COLLECTIVE`` (``"pmix.grp.ftcoll"``, bool, default false); ``grp_notify_termination(blk)`` scans them for ``PMIX_GROUP_NOTIFY_TERMINATION`` (``"pmix.grp.notterm"``, bool, default false). * A **construct without the FT flag** aborts. Every other case survives on the survivors. A **destruct with termination notification** completes cleanly: the ``PMIX_LOCAL_COLLECTIVE_STATUS`` slot is left at ``PMIX_SUCCESS`` and the survivors are told which member was lost via the synthesized ``PMIX_GROUP_MEMBER_FAILED`` event (the event standing in place of an error). Every other survive path — a fault-tolerant **construct**, or a **destruct without notification** — records ``PMIX_ERR_LOST_CONNECTION`` into that slot so the host is told the local phase was degraded (clients still receive their own status through ``grpcbfunc``). ``abort_construct(blk, status)`` deletes any armed timer; if a host is present it issues ``pmix_host_server.group(PMIX_GROUP_CANCEL, blk->id, ...)`` with ``cancel_cbfunc``; sets ``host_called``; and completes the local participants directly by driving ``grpcbfunc`` with the abort status. The ``PMIX_GROUP_CANCEL`` call is essential in the cross-server case: other servers that already forwarded their local phase are waiting on the host collective for a contribution this server will never send, so the host must be told to unstick that collective. Because the block is removed locally, the host's resulting abort release is a no-op back here — no double completion. ``abort_construct`` is only ever called pre-forward, so no host completion can race it. Synthesizing ``PMIX_GROUP_MEMBER_FAILED`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On the *survive* path, ``notify_local_members_of_loss()`` emits, for each departed process, a ``PMIX_GROUP_MEMBER_FAILED`` event carrying ``PMIX_EVENT_AFFECTED_PROC`` (the lost member) and ``PMIX_GROUP_ID``. It is delivered through the internal, progress-thread-safe ``pmix_server_notify_client_of_event`` — never the public ``PMIx_Notify_event``. Because each server does this accounting for its own local survivors, a member lost on another node is reported by that node's server performing the identical logic; the feature therefore needs no cooperation from the host environment. ``_grpcbfunc`` calls it when the operation completed successfully (status is ``PMIX_SUCCESS``), has a non-empty ``departed`` list, and is either a construct (which reaches ``_grpcbfunc`` with success only on the fault-tolerant survive path — an aborted construct arrives with ``PMIX_GROUP_CONSTRUCT_ABORT``) or a destruct for which ``grp_notify_termination()`` reports that ``PMIX_GROUP_NOTIFY_TERMINATION`` was requested. For the destruct survive path the two host-forward points (``pmix_server_group`` and ``account_departed``) deliberately leave the local-collective status at ``PMIX_SUCCESS`` when notification was requested (so the operation returns success, the event standing in place of an error) and record ``PMIX_ERR_LOST_CONNECTION`` otherwise. The local-phase timeout ^^^^^^^^^^^^^^^^^^^^^^^^^ A ``PMIX_TIMEOUT`` directive arms a libevent timer over the local phase via ``PMIX_THREADSHIFT_DELAY(blk, group_timeout, tmo)``, setting ``blk->ev`` and ``event_active``. ``group_timeout()`` fires ``abort_construct(blk, PMIX_ERR_TIMEOUT)``. The timer is deleted before every host-forward (and in the block destructor ``gbdes`` as an error-path safety net), so it can never fire on an already-forwarded — and possibly freed — block. As with the fence family, this timer covers only the *local* phase; the host owns the cross-server timeout. The collective-status slot ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``pmix_server_set_collective_status()`` (in ``src/server/pmix_server_ops.c``) locates the ``PMIX_LOCAL_COLLECTIVE_STATUS`` entry **by key** (not by position) and writes the status in place; an absent slot is a no-op. Finding it by key matters for connect, which appends per-participant and job-level info after the seeded slot — a positional write would corrupt that appended data and leave the real status stale. The same helper is used by the fence-family ``lost_connection`` path, both group host-forward points, and the white-box unit test ``test/unit/collective_status.c``. The host callback boundary -------------------------- Group construct/destruct/cancel all cross the PMIx-to-host boundary through one callback slot, ``pmix_server_module_t.group``, whose operation is a ``pmix_group_operation_t``: .. code-block:: c typedef enum { PMIX_GROUP_CONSTRUCT, /* 0 */ PMIX_GROUP_DESTRUCT, /* 1 */ PMIX_GROUP_NONE, /* 2 */ PMIX_GROUP_CANCEL /* 3 */ } pmix_group_operation_t; Because the enum's numeric values cross that boundary, new values **must** be appended, never inserted — ``PMIX_GROUP_CANCEL`` was added at the end. ``PMIx_Group_operation_string()`` (in ``src/common/pmix_strings.c``) stringifies each value. Two footguns to respect when editing this enum: * Keep the body **free of comment lines**: the Python bindings generator ``bindings/python/construct.py`` treats every line inside a ``typedef enum`` as an enumerator, so an interleaved comment shifts every subsequent value. The explanatory comment therefore lives in a block above the ``typedef``. * ``PMIX_GROUP_CANCEL`` is *not* carried on the client-to-server wire (it is purely a PMIx-to-host callback op), so no new ``bfrops`` support is needed. Capability detection ^^^^^^^^^^^^^^^^^^^^^ The whole feature set above is advertised by the ``PMIX_CAP_GROUP_FT`` capability flag in ``include/pmix_version.h.in``. Its *presence* in the installed ``pmix_version.h`` is the signal a consumer (e.g. PRRTE, via ``PRTE_CHECK_PMIX_CAP``) tests for; absence means the running PMIx predates group fault tolerance. Edit the ``.in`` template, never the generated header. Event and attribute reference ----------------------------- Events (defined in ``include/pmix_common.h.in``): ============================================ ====== ================================================================ Event Value Role ============================================ ====== ================================================================ ``PMIX_GROUP_INVITED`` -159 Leader → invitees; carries ``PMIX_GROUP_ID`` ``PMIX_GROUP_LEFT`` -160 Departing proc → members; server intercepts to account the leave ``PMIX_GROUP_INVITE_ACCEPTED`` -161 Invitee → leader (``PMIx_Group_join`` accept) ``PMIX_GROUP_INVITE_DECLINED`` -162 Invitee → leader (``PMIx_Group_join`` decline) ``PMIX_GROUP_INVITE_FAILED`` -163 Leader-local report of a non-accepter ``PMIX_GROUP_MEMBERSHIP_UPDATE`` -164 Membership change notification ``PMIX_GROUP_CONSTRUCT_ABORT`` -165 Construct aborted; all participants notified ``PMIX_GROUP_CONSTRUCT_COMPLETE`` -166 Group formed; carries final ``PMIX_GROUP_MEMBERSHIP`` ``PMIX_GROUP_LEADER_SELECTED`` -167 App-driven reselection result ``PMIX_GROUP_LEADER_FAILED`` -168 Leader lost; surfaced to a joining member's handlers ``PMIX_GROUP_CONTEXT_ID_ASSIGNED`` -169 Context ID assigned ``PMIX_GROUP_MEMBER_FAILED`` -170 Server-synthesized on a lost member (survive path) ============================================ ====== ================================================================ Key attributes / status codes: * ``PMIX_GROUP_FT_COLLECTIVE`` (``"pmix.grp.ftcoll"``, bool) — the survive-vs-abort gate for a lost member during construct. * ``PMIX_GROUP_OPTIONAL`` (bool) — all-or-nothing (default) vs. reduced group for the invite method. * ``PMIX_GROUP_NOTIFY_TERMINATION`` (bool) — report vs. error a member lost mid-destruct. * ``PMIX_GROUP_LEADER`` (bool) — declares the construct leader; also returned from a leader-failed handler to nominate a replacement. * ``PMIX_LOCAL_COLLECTIVE_STATUS`` (``pmix_status_t``) — the per-tracker health slot the fault machinery updates by key. * ``PMIX_ERR_NOT_A_MEMBER`` — returned when the caller is not among the expanded construct participants. * ``PMIX_ERR_TIMEOUT`` — local-phase timeout expiry. * ``PMIX_ERR_LOST_CONNECTION`` / ``PMIX_ERR_PARTIAL_SUCCESS`` — degraded local-collective status recorded for the host. Testing ------- The behaviors above are exercised at several levels: * ``test/unit/collective_status.c`` — white-box coverage of the by-key status helper (connect/fence layouts, absent slot, degenerate inputs), wired into ``make check``. * ``test/simple`` drivers — ``simpgrpdie`` (construct survives loss + ``PMIX_GROUP_MEMBER_FAILED``, with ``PMIX_GROUP_FT_COLLECTIVE``), ``simpgrpddie`` (destruct survives loss + ``PMIX_GROUP_MEMBER_FAILED`` and returns success, with ``PMIX_GROUP_NOTIFY_TERMINATION``), ``simpgrpcabort`` (default abort with ``PMIX_GROUP_CONSTRUCT_ABORT``), and ``simpgrpctimeout`` (``PMIX_ERR_TIMEOUT`` on a hung live member). * ``examples/`` exercisers driven by the dockerswarm harness (``contrib/dockerswarm/run-group-events.sh``) across a multi-node swarm, so the cross-server notification and cancel paths actually run: ``group_invite`` (happy path), ``group_invite_timeout`` / ``group_invite_decline`` / ``group_invite_abort`` (invite fault paths), ``group_destruct_die`` and ``group_die`` (loss mid-destruct/construct), ``group_construct_abort`` (lost member forces ``PMIX_GROUP_CANCEL`` to unstick remote servers), ``group_daemon_fail`` (whole-daemon loss), and ``group_leave``.