10. The Modular Component Architecture (MCA)

PMIx is a highly-customizable system; it can be configured via configuration files, command line parameters, and environment variables. The main functionality of PMIx’s configuration system is through the Modular Component Architecture (MCA).

  • This section describes the MCA itself and how to set MCA parameters at run time.

  • Later sections in this documentation describe different parts of PMIx’s functionality, and the specific names and values of MCA parameters that can be used to affect PMIx’s behavior.

10.1. Terminology

The Modular Component Architecture (MCA) is the backbone for much of PMIx’s functionality. It is a series of frameworks, components, and modules that are assembled at run-time to create the PMIx implementation.

MCA parameters (also known as MCA variables) are used to customize PMIx’s behavior at run-time.

Each of these entities are described below.

10.1.1. Frameworks

An MCA framework manages zero or more components at run-time and is targeted at a specific task (e.g., providing support for network functionality). Although each MCA framework supports only a single type of component, it may support multiple components of that type.

Some of the more common frameworks that users may want or need to customize include the following:

  • ptl: PMIx Transport Layer; users may need to assist the messaging layer with selection of network interfaces and controls parameters

  • pmdl: Programming Model and Library; identify environmental variables to be forwarded to application processes

There are many frameworks within PMIx; the exact set varies between different versions of PMIx. You can use the pmix_info(1) command to see the full list of frameworks that are included in PMIx latest.

10.1.2. Components

An MCA component is an implementation of a framework’s formal interface. It is a standalone collection of code that can be bundled into a plugin that can be inserted into the PMIx code base, either at run-time and/or compile-time.

Note

Good synonyms for PMIx’s “component” concept are “plugin”, or “add-on”.

The exact set of components varies between different versions of Open MPI. PMIx’s code base includes support for many components, but not all of them may be present or available on your system. You can use the pmix_info(1) command to see what components are included in PMIx latest on your system.

10.1.3. Modules

An MCA module (frequently called a plugin) is an instance of a component. While it is possible for a component to have multiple active plugins, it would be a very rare component that did so.

10.1.4. Parameters (variables)

MCA parameters (sometimes called MCA variables) are the basic unit of run-time tuning for PMIx. They are simple “key = value” pairs that are used extensively throughout PMIx. The general rules of thumb that the developers use are:

  1. Instead of using a constant for an important value, make it an MCA parameter so users can adjust it if necessary.

  2. If a task can be implemented in multiple, user-discernible ways, implement as many as possible, and use an MCA parameter to choose between them at run-time.

For example, the PMIx server sets a maximum value on the number of events it will cache. Events received after that point cause the oldest events in the cache to be released. This allows the server to provide a process with events it registers to receive, even if the registration occurs after the event has been received by the server - thereby reducing the likelihood of a process missing an event due to a race condition. Applications that generate a large volume of events (e.g., an application relying on the PMIx Group invite/join method for creating collections of processes) might require a significantly larger cache than the default size.

10.2. Setting MCA parameter values

MCA parameters may be set in several different ways.

Rationale

Having multiple methods to set MCA parameters allows, for example, system administrators to fine-tune the PMIx installation for their hardware / environment such that normal users can simply use the default values (that were set by the system administrators).

HPC environments — and the applications that run on them — tend to be unique. Providing extensive run-time tuning capabilities through MCA parameters allows the customization of PMIx to each system’s / user’s / application’s particular needs.

The following are the different methods to set MCA parameters, listed in priority order:

  1. Environment variables

  2. Configuration files

10.2.1. Environment variables

Environment variables are given the highest priority. Any environment variable named PMIX_MCA_<param_name> will be examined for use. Note that misspelling of names will cause the value to be ignored - i.e., PMIx does not report unrecognized MCA parameters, it simply ignores them.

Note

Just like with command line values, setting environment variables to values with multiple words requires shell quoting, such as:

shell$ export OMPI_MCA_param="value with multiple words"

10.2.2. Configuration files

Simple configuration text files can also be used to set MCA parameter values. Parameters are set one per line (comments are permitted). For example:

# This is a comment
# Set an MCA parameter
mca_component_show_load_errors = 1

Note that quotes are not necessary for setting multi-word values in MCA parameter files. Indeed, if you use quotes in the MCA parameter file, they will be used as part of the value itself. For example:

# The following two values are different:
param1 = value with multiple words
param2 = "value with multiple words"

By default, two files are searched (in order):

  1. $HOME/.pmix/mca-params.conf: The user-supplied set of values has the higher precedence.

  2. $prefix/etc/pmix-mca-params.conf: The system-supplied set of values has a lower precedence.

More specifically, the MCA parameter mca_param_files specifies a colon-delimited path of files to search for MCA parameters. Files to the left have lower precedence; files to the right are higher precedence.

Note

Keep in mind that, just like components, these parameter files are only relevant where they are “visible”. Typically, these files are read by the host daemon responsible for launching an application and then forwarded to all daemons (and their child application processes) in their environment.

Warning

Setting PMIx MCA parameters via configuration files entails editing (by default) the following files:

$HOME/.pmix/mca-params.conf or $prefix/etc/pmix-mca-params.conf

10.3. Selecting which PMIx components are used at run time

Each MCA framework has a top-level MCA parameter that helps guide which components are selected to be used at run-time. Specifically, every framework has an MCA parameter of the same name that can be used to include or exclude components from a given run.

For example, the pmdl MCA parameter can used to control which PMDL components are used. It takes a comma-delimited list of component names, and may be optionally prefixed with ^. For example:

Note

The Programming Model PMDL framework provides support for a range of programming models and libraries, including collection of default parameters and environmental variables for forwarding and setting of library-specific environmental variables

# Tell PMIx to include *only* the PMDL components listed here and
# implicitly ignore all the rest:
export PMIX_MCA_pmdl=ompi,oshmem ...

# Tell PMIx to exclude the ompi and oshmem PMDL components
# and implicitly include all the rest
export PMIX_MCA_pmdl=^ompi,oshmem ...

Note that ^ can only be the prefix of the entire comma-delimited list because the inclusive and exclusive behavior are mutually exclusive. Specifically, since the exclusive behavior means “use all components except these”, it does not make sense to mix it with the inclusive behavior of not specifying it (i.e., “use all of these components”). Hence, something like this:

export PMIX_MCA_pmdl=ompi,^oshmem ...

does not make sense — and will cause an error — because it says “use only the ompi component” but also “use all components except oshmem”. These two statements clearly contradict each other.

10.4. Common MCA parameters

PMIx has a large number of MCA parameters available. Users can use the pmix_info(1) command to see all available MCA parameters.

The vast majority of these MCA parameters, however, are not useful to most users. Although the full list of MCA parameters can be found in the output of pmix_info(1), the following list of commonly-used parameters is presented here so that they can easily be found via internet searches:

  • Individual framework names with the _base_verbose suffix appended (e.g., ptl_base_verbose, pmdl_base_verbose, etc.) can be used to set the general verbosity level of all the components in that framework.

    • This can be helpful when troubleshooting why certain components are or are not being selected at run time.

  • The PMIx Transport Layer supports “include” and “exclude” types of components (e.g., ptl_tcp_if_include and ptl_tcp_if_exclude). The “include” parameters specify an explicit set of network interfaces to use; the “exclude” parameters specify an explicit set of network interfaces to ignore. Check the output from pmix_info(1)’s to see the full list of PTL-related parameters.

    Important

    You can only use the “include” or the “exclude” parameter — they are mutually exclusive from each other.

  • mca_base_component_show_load_errors: By default, PMIx emits a warning message if it fails to open a DSO component at run time. This typically happens when a shared library that the DSO requires is not available.

    Rationale

    In prior versions of PMIx, components defaulted to building as DSOs (vs. being included in the parent library, libpmix.so). On misconfigured systems, sometimes libraries required by various components would not be present, thereby causing those components to fail to open at run time.

    Having PMIx warn about such failures to load was useful because it alerted users to the misconfiguration.

    Note

    By default, PMIx latest includes all components in its base libraries (e.g., on Linux, libpmix.so includes all the components that were built with PMIx, and therefore no component need to be opened dynamically), and does not build its components as DSOs.

    This MCA parameter only affects the behavior when a component DSO fails to open.

    This MCA parameter can take four general values:

    1. yes or a boolean “true” value (e.g., 1): PMIx will emit a warning about every component DSO that fails to load.

    2. no or a boolean “false” value (e.g., 0): PMIx will never emit warnings about component DSOs that fail to load.

    3. A comma-delimited list of frameworks and/or components: PMIx will emit a warning about any dynamic component that fails to open and matches a token in the list. “Match” is defined as:

      • If a token in the list is only a framework name, then any component in that framework will match.

      • If a token in the list specifies both a framework name and a component name (in the form framework/component), then only the specified component in the specified framework will match.

      For example, if the value of this MCA parameter is pmdl,pnet/opa, then PMIx will warn if any component in the PMDL framework or if the OPA PNET component fails to load at run time.

    4. The value can also be a ^ character followed by a comma-delimited list of framework[/component] values: This is similar to the comma-delimited list of tokens, except it will only emit warnings about dynamic components that fail to load and do not match a token in the list.

      For example, if the value of this MCA parameter is ^pmdl,pnet/opa, then PMIx will only warn about the failure to load DSOs that are neither in the PMDL framework nor are the OPA PNET component.