`srun` with OpenMPI 5.0.3 unexpectedly launches MPI jobs as singletons without ERROR

[GayatriManda]

As an OpenMPI user, I noticed unexpected behavior when running MPI programs with Slurm’s srun.

• Environment: module load OpenMPI/5.0.3
• What happens:

Using mpirun (works as expected)

Hello from proc 0 of 4
Hello from proc 1 of 4
Hello from proc 2 of 4
Hello from proc 3 of 4

Using srun --mpi=pmi2

No PMIx server was reachable, but a PMI1/2 was detected.
If srun is being used to launch application, 4 singletons will be started.
Hello from proc 0 of 1
Hello from proc 0 of 1
Hello from proc 0 of 1
Hello from proc 0 of 1

Using plain srun (without explicitly mentioning --mpi)

Hello from proc 0 of 1
Hello from proc 0 of 1
Hello from proc 0 of 1
Hello from proc 0 of 1

• Why this is a problem: With --mpi=pmi2, OpenMPI atleast prints a runtime information before falling back to singleton mode. With plain srun, the same fallback happens but no warning is shown. As a user, this is very misleading: the job looks like a normal MPI run, but every process starts as a singleton rank 0 of 1, so no communication happens and resources are wasted.

• What I would expect: It would be more helpful if OpenMPI issued an error or warning whenever it cannot connect to PMI/PMIx under srun, rather than silently launching singletons, and showed the same warning even when --mpi= is not explicitly specified.

This would prevent us from unintentionally running incorrect MPI jobs.

[rhc54]

Errr...what you describe is indeed accurate. However, adding Slurm-specific detection gets dicey - we would have to add detection for every resource manager out there. Doable, I suppose - but a pretty big ask and very hard to maintain (e.g., Slurm constantly changes their envars between versions - we've chased them for years for other purposes with marginal success).

If someone feels it should be done and wants to do it, they are welcome to do so.