docs: numerous updates to MPI_Init*/MPI_Finalize*/MPI_Session_* man pages

Including, but not limited to:

* Added much more description of and distinction between the MPI world
  model and the MPI session model.  Updated a lot of old,
  pre-MPI-world-model/pre-MPI-session-model text that was now stale /
  outdated, especially in the following pages:
  * MPI_Init(3), MPI_Init_thread(3)
  * MPI_Initialized(3)
  * MPI_Finalize(3)
  * MPI_Finalized(3)
  * MPI_Session_init(3)
  * MPI_Session_finalize(3)
* Numerous formatting updates
* Slightly improve the C code examples
* Describe the mathematical relationship between the various
  MPI_THREAD_* constants in MPI_Init_thread(3)
  * Note that the mathematical relationships render nicely in HTML,
    but don't render entirely properly in nroff.  This commit author
    is of the opinion that the nroff rendering is currently "good
    enough", and some Sphinx maintainer will fix it someday.
* Add descriptions about the $OMPI_MPI_THREAD_LEVEL env variable and
  how it is used in MPI_Init_thread(3)
* Added more seealso links

Signed-off-by: Jeff Squyres <jeff@squyres.com>

Author: jsquyres

docs/man-openmpi/man3/MPI_Finalize.3.rst

@@ -5,7 +5,7 @@ MPI_Finalize
 
 .. include_body
 
-:ref:`MPI_Finalize` |mdash| Terminates MPI execution environment.
+:ref:`MPI_Finalize` |mdash| Terminates MPI world model.
 
 .. The following file was automatically generated
 .. include:: ./bindings/mpi_finalize.rst
@@ -18,56 +18,82 @@ OUTPUT PARAMETER
 DESCRIPTION
 -----------
 
-This routine cleans up all MPI states. Once this routine is called, no
-MPI routine (not even MPI_Init) may be called, except for
-:ref:`MPI_Get_version`, :ref:`MPI_Initialized`, and :ref:`MPI_Finalized`. Unless there has
-been a call to :ref:`MPI_Abort`, you must ensure that all pending
-communications involving a process are complete before the process calls
-:ref:`MPI_Finalize`. If the call returns, each process may either continue
-local computations or exit without participating in further
-communication with other processes. At the moment when the last process
-calls :ref:`MPI_Finalize`, all pending sends must be matched by a receive, and
-all pending receives must be matched by a send.
-
-:ref:`MPI_Finalize` is collective over all connected processes. If no processes
-were spawned, accepted, or connected, then this means it is collective
-over MPI_COMM_WORLD. Otherwise, it is collective over the union of all
-processes that have been and continue to be connected.
+This routine finalizes the MPI world model.  If the MPI world model
+has been initialized in an MPI process, it *must* be finalized exactly
+once by invoking this routine during the lifetime of that MPI process.
+This is different than the MPI session model, which can be initialized
+and finalized multiple times in an MPI process.  See
+:ref:`MPI_Session_init` and :ref:`MPI_Session_finalize`.
+
+Unless there has been a call to :ref:`MPI_Abort`, you must
+ensure that all pending communications in the MPI world model
+involving a process are complete before the process calls
+:ref:`MPI_Finalize`. If the call returns, each process may either
+continue local computations or exit without participating in further
+communication with other processes in the MPI world model. At the
+moment when the last process calls :ref:`MPI_Finalize`, all pending
+sends in the MPI world model must be matched by a receive, and all
+pending receives in the MPI world model must be matched by a send.
+
+See `MPI-5.0:11.4.1 <https://www.mpi-forum.org/>`_ for a list of MPI
+functionality that is available (e.g., even when the MPI
+world model has not yet initialized or has already been finalized).
+
+:ref:`MPI_Finalize` is collective over all connected processes. If no
+processes were spawned, accepted, or connected, then this means it is
+collective over ``MPI_COMM_WORLD``. Otherwise, it is collective over
+the union of all processes that have been and continue to be
+connected.
 
 NOTES
 -----
 
-All processes must call this routine before exiting. All processes will
-still exist but may not make any further MPI calls. :ref:`MPI_Finalize`
-guarantees that all local actions required by communications the user
-has completed will, in fact, occur before it returns. However,
-:ref:`MPI_Finalize` guarantees nothing about pending communications that have
-not been completed; completion is ensured only by :ref:`MPI_Wait`, :ref:`MPI_Test`, or
-:ref:`MPI_Request_free` combined with some other verification of completion.
-
-For example, a successful return from a blocking communication operation
-or from :ref:`MPI_Wait` or :ref:`MPI_Test` means that the communication is completed
-by the user and the buffer can be reused, but does not guarantee that
-the local process has no more work to do. Similarly, a successful return
-from :ref:`MPI_Request_free` with a request handle generated by an :ref:`MPI_Isend`
-nullifies the handle but does not guarantee that the operation has
-completed. The :ref:`MPI_Isend` is complete only when a matching receive has
-completed.
-
-If you would like to cause actions to happen when a process finishes,
-attach an attribute to MPI_COMM_SELF with a callback function. Then,
-when :ref:`MPI_Finalize` is called, it will first execute the equivalent of an
-:ref:`MPI_Comm_free` on MPI_COMM_SELF. This will cause the delete callback
-function to be executed on all keys associated with MPI_COMM_SELF in an
-arbitrary order. If no key has been attached to MPI_COMM_SELF, then no
-callback is invoked. This freeing of MPI_COMM_SELF happens before any
-other parts of MPI are affected. Calling :ref:`MPI_Finalized` will thus return
-"false" in any of these callback functions. Once you have done this with
-MPI_COMM_SELF, the results of :ref:`MPI_Finalize` are not specified.
+The MPI session model is different than the MPI world model, and has
+different scopes of availability for MPI functionality.  See
+:ref:`MPI_Session_init` and :ref:`MPI_Session_finalize`.
+
+All processes that initialized the MPI world model must call this
+routine before exiting.  All processes will still exist but may not
+make any further MPI calls in the MPI world model. :ref:`MPI_Finalize`
+guarantees that all local actions required by communications in the
+MPI world model that the user has completed will, in fact, occur
+before it returns. However, :ref:`MPI_Finalize` guarantees nothing
+about pending communications in the MPI world model that have not been
+completed; completion is ensured only by the :ref:`MPI_Wait` and
+:ref:`MPI_Test` variants, or :ref:`MPI_Request_free` combined with
+some other verification of completion.
+
+For example, a successful return from a blocking communication
+operation or from one of the :ref:`MPI_Wait` or :ref:`MPI_Test`
+varients means that the communication is completed by the user and the
+buffer can be reused, but does not guarantee that the local process
+has no more work to do. Similarly, a successful return from
+:ref:`MPI_Request_free` with a request handle generated by an
+:ref:`MPI_Isend` nullifies the handle but does not guarantee that the
+operation has completed. The :ref:`MPI_Isend` is complete only when a
+matching receive has completed.
+
+If you would like to cause actions to happen when a process finalizes the MPI
+world model, attach an attribute to ``MPI_COMM_SELF`` with a callback
+function. Then, when :ref:`MPI_Finalize` is called, it will first
+execute the equivalent of an :ref:`MPI_Comm_free` on
+``MPI_COMM_SELF``. This will cause the delete callback function to be
+executed on all keys associated with ``MPI_COMM_SELF`` in an arbitrary
+order. If no key has been attached to ``MPI_COMM_SELF``, then no
+callback is invoked. This freeing of ``MPI_COMM_SELF`` happens before
+any other parts of the MPI world model are affected. Calling
+:ref:`MPI_Finalized` will thus return ``false`` in any of these
+callback functions. Once you have done this with ``MPI_COMM_SELF``,
+the results of :ref:`MPI_Finalize` are not specified.
 
 ERRORS
 ------
 
 .. include:: ./ERRORS.rst
 
-.. seealso:: :ref:`MPI_Init`
+.. seealso::
+   * :ref:`MPI_Finalized`
+   * :ref:`MPI_Init`
+   * :ref:`MPI_Initialized`
+   * :ref:`MPI_Session_init`
+   * :ref:`MPI_Session_finalize`

docs/man-openmpi/man3/MPI_Finalized.3.rst

@@ -5,28 +5,39 @@ MPI_Finalized
 
 .. include_body
 
-:ref:`MPI_Finalized` |mdash| Checks whether MPI has been finalized
+:ref:`MPI_Finalized` |mdash| Checks whether the MPI world model has been finalized
 
 .. The following file was automatically generated
 .. include:: ./bindings/mpi_finalized.rst
 
 OUTPUT PARAMETER
 ----------------
 
-* ``flag`` : True if MPI was finalized, and false otherwise (logical).
+* ``flag`` : True if the MPI world model was finalized, and false
+  otherwise (logical).
 * ``ierror`` : Fortran only: Error status (integer).
 
 DESCRIPTION
 -----------
 
-This routine may be used to determine whether MPI has been finalized. It
-is one of a small number of routines that may be called before MPI is
-initialized and after MPI has been finalized (:ref:`MPI_Initialized` is
-another).
+This routine may be used to determine whether the MPI world model has
+been finalized.  A different routine |mdash| :ref:`MPI_Initialized`
+|mdash| is used to indicate whether the MPI world model has been
+initialized.
+
+See `MPI-5.0:11.4.1 <https://www.mpi-forum.org/>`_ for a list of MPI
+functionality that is available (e.g., even when the MPI
+world model has not yet initialized or has already been finalized).
 
 ERRORS
 ------
 
 .. include:: ./ERRORS.rst
 
-.. seealso:: :ref:`MPI_Init`
+.. seealso::
+   * :ref:`MPI_Init`
+   * :ref:`MPI_Init_thread`
+   * :ref:`MPI_Finalize`
+   * :ref:`MPI_Finalized`
+   * :ref:`MPI_Session_init`
+   * :ref:`MPI_Session_finalize`

docs/man-openmpi/man3/MPI_Init.3.rst

@@ -6,7 +6,7 @@ MPI_Init
 
 .. include_body
 
-:ref:`MPI_Init` |mdash| Initializes the MPI execution environment
+:ref:`MPI_Init` |mdash| Initializes the MPI world model
 
 .. The following file was automatically generated
 .. include:: ./bindings/mpi_init.rst
@@ -23,35 +23,53 @@ OUTPUT PARAMETER
 DESCRIPTION
 -----------
 
-This routine, or :ref:`MPI_Init_thread`, must be called before most other MPI
-routines are called. There are a small number of errors, such as
-:ref:`MPI_Initialized` and :ref:`MPI_Finalized`. MPI can be initialized at most once;
-subsequent calls to :ref:`MPI_Init` or :ref:`MPI_Init_thread` are erroneous.
+This routine, or :ref:`MPI_Init_thread`, initializes the MPI world
+model.  Either of these routines must be called before MPI
+communication routines are called within the MPI world model.  The MPI
+world model can be initialized at most exactly once in the lifetime of
+an MPI process.  This is different than the MPI session model, which
+can be initialized and finalized multiple times in an MPI process.
+See :ref:`MPI_Session_init` and :ref:`MPI_Session_finalize`.
 
-All MPI programs must contain a call to :ref:`MPI_Init` or :ref:`MPI_Init_thread`.
-Open MPI accepts the C *argc* and *argv* arguments to main, but neither
-modifies, interprets, nor distributes them:
+See `MPI-5.0:11.4.1 <https://www.mpi-forum.org/>`_ for a list of MPI
+functionality that is available (e.g., even when the MPI
+world model has not yet initialized or has already been finalized).
+
+Open MPI's :ref:`MPI_Init` and :ref:`MPI_Init_thread` both accept the
+C *argc* and *argv* arguments to main, but neither modifies,
+interprets, nor distributes them:
 
 .. code-block:: c
 
-   /* declare variables */
-   MPI_Init(&argc, &argv);
-   /* parse arguments */
-   /* main program */
-   MPI_Finalize();
+   #include <mpi.h>
+
+   int main(int argv, char *argv[]) {
+       MPI_Init(&argc, &argv);
+       /* ...body of main MPI pogram... */
+       MPI_Finalize();
+       return 0;
+   }
 
+By default, :ref:`MPI_Init` is effectively equivalent to invoking
+:ref:`MPI_Init_thread` with a *required* value of
+``MPI_THREAD_SINGLE``.  However, if the ``OMPI_MPI_THREAD_LEVEL``
+environment variable is set to a valid value when :ref:`MPI_Init` is
+invoked, it is equivalent to invoking :ref:`MPI_Init_thread` with
+*required* set to the corresponding value of the ``OMPI_MPI_THREAD_LEVEL``
+environment variable.  See :ref:`MPI_Init_thread` for more details.
 
 NOTES
 -----
 
 The Fortran version does not have provisions for *argc* and *argv* and
 takes only IERROR.
 
-The MPI Standard does not say what a program can do before an :ref:`MPI_Init`
-or after an :ref:`MPI_Finalize`. In the Open MPI implementation, it should do
-as little as possible. In particular, avoid anything that changes the
-external state of the program, such as opening files, reading standard
-input, or writing to standard output.
+The MPI Standard does not specify what a program using the MPI world
+model can do before invoking :ref:`MPI_Init` or :ref:`MPI_Init_thread`
+or after invoking :ref:`MPI_Finalize`. In the Open MPI implementation,
+it should do as little as possible. In particular, avoid anything that
+changes the external state of the program, such as opening files,
+reading standard input, or writing to standard output.
 
 
 ERRORS
@@ -64,3 +82,5 @@ ERRORS
    * :ref:`MPI_Initialized`
    * :ref:`MPI_Finalize`
    * :ref:`MPI_Finalized`
+   * :ref:`MPI_Session_finalize`
+   * :ref:`MPI_Session_init`