19. Access control by user and group

Note

This is a plan. It describes the access model PMIx is moving to, and the order in which the pieces are being implemented. The Status section at the end says which parts are in place today. Anything not yet implemented behaves as it did before this work began.

This page is for two audiences:

Users and administrators, who need to know who can see a job’s data and act on a job, and how to widen that.
Developers of PMIx and of host environments, who need to know where each check is made and what a host has to do.

19.1. Summary

Access to a job - reading its data, pulling its output, monitoring or signalling its processes - is decided by who is asking (their user ID and groups), not by which job the requester belongs to.
By default, only the job’s owner can access it. The owner can widen that when the job is started, by naming other users and/or groups with PMIX_ACCESS_PERMISSIONS.
Crossing jobs is normal and supported. A debugger or tool that asks about another job succeeds whenever its user is allowed.
These rules govern requests that clients and tools make of a PMIx server. They do not restrict the host environment itself - see below.

19.2. Who is governed: the host versus clients and tools

A PMIx server lives inside a host environment: a resource manager, launcher, or other system daemon that calls PMIx_server_init and supplies the pmix_server_module_t up-calls. Many hosts run as a privileged service account that is not root.

The host is not restricted. The library does not limit what the host can do through the server APIs (registering jobs, delivering output, answering data requests, and so on), and does not second-guess what the host does with a request passed up to it. The host is the authority for its own operations.
Clients and tools are governed. A request a client or tool sends to a server is checked against the target job’s permissions:
- by the server library, for data and operations it handles itself;
- by the host, for operations the library passes up to it. For these the library tells the host exactly who is asking - the requester’s process ID, PMIX_USERID and PMIX_GRPID (see pmix_server_module_t(5)) - and the host applies the same rule.

The requester’s identity is the one established when it connected to the server (the connection handshake verifies it against the host’s registration). It is not something the request itself can claim.

A request can pass through more than one server: a host relays what one of its clients or tools asked for to another server - a workflow manager to the resource manager, a launcher to a scheduler. The identity is fixed where the request entered, and a relaying host passes it on with its PMIX_USERID and PMIX_GRPID marked PMIX_INFO_RELAYED (PMIx_Info_relayed(3)). The next server keeps them, so the operation is judged by who asked for it rather than by the relaying process. Only a PMIx server’s library sends the mark - any other process’s library clears it - so a process that is not a server speaks only for itself. A server trusts the account a relaying server gives.

19.3. Terms

Owner: The user (and primary group) a job belongs to - normally the user who started it.
Requester: The client or tool making a request, identified by the user ID and group ID recorded when it connected.
Access list: Additional users and groups the owner has allowed, given with PMIX_ACCESS_PERMISSIONS.

19.4. The rule

A requester may access a job if any of the following holds:

the requester’s user ID is 0 (root);
the requester’s user ID is the server’s own effective user ID - the account the host runs as, which can already inspect the server process itself;
the requester’s user ID is the job owner’s;
the requester’s user ID is in the job’s access list;
the requester belongs to a group in the job’s access list - its primary group or any supplementary group.

Otherwise the request fails with PMIX_ERR_NO_PERMISSIONS.

19.5. Establishing a job’s permissions

The owner. The host gives the job’s owner when it registers the job, as PMIX_USERID and PMIX_GRPID in the info passed to PMIx_server_register_nspace. Either may be a number or a user or group name; names are resolved when the job is registered. If the host does not give them:

the owner is the user and group the job’s clients were registered with (PMIx_server_register_client);
failing that, the owner is the server’s own user.

A tool’s owner is the user and group it presented when it connected.

The access list. A user widens access when starting a job, by passing PMIX_ACCESS_PERMISSIONS in the job-level directives of PMIx_Spawn. Its value is a data array of pmix_info_t holding either or both of:

PMIX_ACCESS_USERIDS - a data array of user IDs (numbers or names);
PMIX_ACCESS_GRPIDS - a data array of group IDs (numbers or names).

The host passes the list on when it registers the job, on every node that hosts a part of it. The same attributes already govern access to published data (PMIx_Publish), so users meet one vocabulary.

Group membership is the groups a user belongs to by its account - its primary group and its supplementary groups - and not the group a process happens to be running as. The server keeps a record of each user it knows, keyed by user ID. The host registers each user (PMIx_server_register_resources with PMIX_USERID), and may give the groups it belongs to with it (PMIX_GRPID, as an array), in which case the server looks nothing up. Otherwise the server looks the groups up once, when the user becomes known: the host registers it or a job it owns, or it connects as a tool. Root and the server’s own user are never recorded - they need no groups. A requester the server was never told about - one on another node (see Data held on another node) - is recorded at its first check. A check that finds the user in none of the job’s groups looks them up again before refusing, so a user added to a group is admitted from its next request. A record is dropped when the host deregisters that user (PMIx_server_deregister_resources with PMIX_USERID), and at finalize.

19.6. What is covered

Area	Examples	Checked by
Job-level data	`PMIx_Get` of another job’s data (rank `PMIX_RANK_WILDCARD`)	server library
Process data on this node	`PMIx_Get` of a process’s committed (modex) data	server library
Process data on another node	`PMIx_Get` answered by direct modex	the server that holds the data (see below)
Query and resolve	process tables, namespace information, `PMIx_Resolve_peers`	server library for what it answers; host for the rest
Output forwarding	`PMIx_IOF_pull`, and a pull following a job into the jobs it spawns	host approves each pull; library for inheritance
Output files	`PMIX_IOF_OUTPUT_TO_FILE` / `_TO_DIRECTORY`	file ownership and mode (see below)
Monitoring	`PMIx_Process_monitor` targets, process statistics	server library for processes on its node; host for others
Job control	signal, kill, terminate, abort, stdin	host; server library for cleanup directives it handles
Sessions and allocations	extending, releasing, spawning into an allocation	host (by user, not by namespace)

19.7. What the server library checks

Beyond the rule itself, a few things hold for every check the server library makes:

A job’s own processes always access their own job.
The server’s own namespace describes the server, not anyone’s job, and anyone may read it.
A job the host has not registered with this server - one known here only because data for it arrived from elsewhere - has no permissions here to judge by. Requests for its data are checked by the server that holds it (see Data held on another node). Acting on such a job - cleanup, monitoring, following its output - is allowed only for root and the server’s own user.

Where the checks are made:

Request	Check
`PMIx_Get`, and a refresh of the client’s cached copy	The requester must be allowed the target job. A request waiting for a job to be registered is checked again once it is.
`PMIx_Query_info` naming a namespace (`PMIX_NSPACE`)	The requester must be allowed that job.
`PMIx_Resolve_peers`, `PMIx_Resolve_nodes`	Naming a job, the requester must be allowed it. Naming none, the answer includes only the jobs the requester may access.
`PMIx_Process_monitor` of processes on this node	Processes named by `PMIX_MONITOR_TARGET_PROCS` must all belong to jobs the requester may access, or the request is refused; each is read at the process ID its host recorded (`PMIX_PROC_PID`). Processes named by `PMIX_MONITOR_TARGET_PIDS` must all be owned by the requester, and -1 means every process it owns. A request for every local process leaves out those the requester may not access. Requests from the host are not restricted, except one it relays from another node (`PMIX_MONITOR_PROXY`), which must name the requester’s `PMIX_USERID` and is checked for that user, and is refused without it. Each process must still belong to the expected user when it is read.
Cleanup directives (`PMIX_REGISTER_CLEANUP` and its family) on `PMIx_Job_control`	The requester must be allowed every job it names, and the cleanup runs as the requester - each entry records who asked - never as the job’s user or the server. A job the server does not know is refused.
Output forwarding inherited by a spawned job	A subscription to a job’s output follows into the jobs it spawns only if its requester may access each of them.

19.8. Data held on another node

When a requester asks for data the local server does not have, the server asks its host, which fetches it from the server that holds it (direct modex). The check is made by the server that holds the data, because that server has the job’s permissions:

The requesting server names the requester in its direct_modex up-call - PMIX_REQUESTOR, PMIX_USERID and PMIX_GRPID, the user and group being those established when the requester connected. Requesters of different identities are sent up separately; requesters of one identity share a fetch.
The host passes that identity to the holding server through PMIx_server_dmodex_request2, a new form of PMIx_server_dmodex_request that takes an info array (every PMIx API carries one).
The holding server applies the rule and answers, or refuses with PMIX_ERR_NO_PERMISSIONS.
The requesting server keeps a copy of what it receives, and remembers which identities the holding server approved. From that copy it answers only those, the job’s own processes, root, and its own user; any other requester is asked about as though the copy were not there, and gets its own fetch. This applies to PMIx_Get, to a refresh of a client’s cache, and to query and resolve answered from the copy.

A host that still uses PMIx_server_dmodex_request passes no requester identity, and the holding server answers as it does today. The capability flag PMIX_CAP_DMODEX_REQUEST2 tells a host that the library provides PMIx_server_dmodex_request2.

19.9. Output files and shared memory

Output files (PMIX_IOF_OUTPUT_TO_FILE, PMIX_IOF_OUTPUT_TO_DIRECTORY). By default only the job’s owner reads them:

When the file belongs to the owner - the server runs as the owner, or runs as root and gives the file to them - it is created with mode 0600, and the directories the server creates for it 0700.
If the access list names groups, the files are made readable by the first listed group (0640, directories 0750), where the server can set that group. File modes can express only an owner and one group, so further users and groups in the list are not applied to files. Access control lists are not used, as they are not portable.
A server that cannot give the file to the owner - one running as a service account other than root - makes it readable by the owner’s own group (0640) if it can set that group, and otherwise leaves it readable by everyone (0644), so the owner can still read their output. Its directories are then 0755: they must let the owner reach the files, whose own modes decide who reads them.
An output file that is not a regular file with a single name is not written.

Shared-memory job data (gds/shmem3). Only the server writes a segment: clients open its backing file read-only and map it read-only. The file belongs to the job’s owner, with mode 0400 - 0440 for the first group the access list names - and never grants write access. A client the mode does not admit - of another user’s job, or of a job whose file the server could not give to its owner - receives the job’s data from the server instead, where the rule is applied.

19.10. Not covered

Event notification. The process raising an event chooses who receives it through the event’s range. The one exception is PMIX_DEBUGGER_RELEASE, which lets a process held for a debugger continue: a server accepts it from a client or tool only if the rule above permits that requester for the job of every process the event can reach. A job the server has not had registered is left to the host, which relays the event.
Published data (PMIx_Publish / PMIx_Lookup). The host’s data server already governs it per published item with PMIX_ACCESS_PERMISSIONS.

19.11. Compatibility

A host that registers no owner gets the fallback above: the user its clients were registered with, then the server’s own user. On a node that hosts none of a job’s processes there are no clients, so there only root and the server’s own user can access that job. A host running as the same user as its jobs sees no change; a host running as a service account should register each job’s owner.
A host that does not pass ``PMIX_ACCESS_PERMISSIONS`` at registration gets owner-only access for every job, which is the default.
Clients and tools of any version are identified by their connection, so nothing changes for them.
Hosts can detect the library’s support through capability flags in pmix_version.h: PMIX_CAP_REQUESTER_ID (the requester’s identity on up-calls), PMIX_CAP_DMODEX_REQUEST2 (PMIx_server_dmodex_request2) and PMIX_CAP_ACCESS_CHECK (pmix_server_access_check()).

19.12. What a host needs to do

Register each job’s owner (PMIX_USERID, PMIX_GRPID) and access list (PMIX_ACCESS_PERMISSIONS) with PMIx_server_register_nspace on every node that hosts part of the job.
Accept PMIX_ACCESS_PERMISSIONS in spawn requests and carry it with the job.
Apply the rule to the operations it performs for clients and tools - job control, abort, stdin, IOF pull approval, queries it answers, monitoring of other nodes, sessions and allocations - using the requester identity the library passes in each up-call. A host linked with the library applies exactly the library’s rule by keeping its own copies and calling pmix_server_access_check() (declared in the installed src/server/pmix_server_ops.h) from its own thread: a pmix_access_t for each job, loaded with pmix_server_access_load() from the info it registers the job with, and a pmix_user_t for each requester. None of these touch the library’s state. When the host drops a user it deregisters it from the library too, so the two stay in step. The PMIX_GRPID in an up-call is the group the requester chose to charge the work to and is not verified: it is not an input to the rule.
Carry the requester’s identity when it relays a request to another of its daemons.
Answer direct-modex requests with PMIx_server_dmodex_request2, passing the requester’s identity.

PRRTE is the reference host, and implements these alongside the library changes.

19.13. Status

Phase	Work	State
0	The requester’s process ID, `PMIX_USERID` and `PMIX_GRPID` are passed on every up-call made for a client or tool; `PMIX_REQUESTOR` where an up-call has no process argument; user and group names accepted; IOF pulls forwarded only after the host approves. PRRTE approves an IOF pull only for root, the DVM’s user, or the job’s owner.	Done
1	Each job’s owner and access list stored by the server library, from the host’s registration; one check implementing the rule; group membership cached per user.	Done
2	The server library applies the rule to what it answers itself: job and process data, query and resolve, monitoring of local processes, cleanup directives, and IOF inheritance.	Done
3	`PMIx_server_dmodex_request2`; the requester’s identity on the `direct_modex` up-call; the check at the holding server; the requesting server answers from its copy only the requesters the holder approved. PRRTE registers each job’s owner (`PMIX_USERID`, `PMIX_GRPID`) with `PMIx_server_register_nspace`, carries the requester’s identity on its direct-modex relay, and uses `PMIx_server_dmodex_request2`.	Done
4	Output file ownership and modes; server-mediated job data for requesters the shared-memory file mode does not admit, with shared-memory segments read-only to their readers.	Done
5	`pmix_server_access_check()`, the rule as a function a host calls with its own copies of a job’s access list and the requester’s groups; users registered and deregistered by the host; group membership looked up again before refusing; a monitor request relayed from another node checked for the requester it names. PRRTE carries each job’s access list (`--rtos users=,groups=`, and `PMIX_ACCESS_PERMISSIONS` at spawn) and applies the rule to the operations it performs.	Done