Warn users if mpi4py running under different MPI implementation
See original GitHub issueIt does happen to unexperienced users that they run pacakges using mpi4py
under an MPI implementation different from the one that mpi4y was built with.
MPI.jl, the analogue of mpi4py in Julia land, tries to warn users if this happens by detecting at MPI_INIT
time if MPI_COMM_WORLD
reports only 1 rank but the known environment variables (MPI_LOCALNRANKS
, OMPI_COMM_WORLD_SIZE
) suggest that there should be more than 1 rank. This happens exactly when mpi4py
would be built with mpich but run under open-mpi.
If this condition is detected it prints a warning.
What do you think about doing this? Is it something that you would merge into mpi4py?
Issue Analytics
- State:
- Created 2 years ago
- Comments:19 (13 by maintainers)
Top Results From Across the Web
Installation — MPI for Python 3.1.4 documentation
If you want to reinstall the mpi4py package using a different or updated MPI implementation, you have to either first remove the cached...
Read more >Tutorial — MPI for Python 3.1.4 documentation
The book covers parallel programming with MPI and OpenMP in C/C++ and Fortran, and MPI in Python using mpi4py. MPI for Python supports...
Read more >mpi4py.futures — MPI for Python 3.1.4 documentation
This package provides a high-level interface for asynchronously executing callables on a pool of worker processes using MPI for inter-process communication.
Read more >Installation — MPI for Python 4.0.0.dev0 documentation
mpi4py supports two different build backends: setuptools (default) and scikit-build. ... If the MPI implementation does not provide a compiler wrapper, ...
Read more >Overview — MPI for Python 3.1.4 documentation
This interface is a standard Python mechanism provided by some types (e.g., strings and numeric arrays), allowing access in the C side to...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@leofang These envars are set by the
mpiexec
command in each process it executes, and then the MPI library takes over to assign a rank toMPI_COMM_WORLD
. If mpiexec is not used (or one from a different implementation is used), then the expected environ vars are not set and the MPI library assumes a singleton MPI_Init() case, thus bootstrapping a sequential run (world size 1).If there is a mismatch between the size of
MPI_COMM_WORLD
and these environment variables, you can reasonably assume that users are messing up: thempiexec
command does not match the MPI library. Did I make myself clear enough or should I provide a specific example using the command line?As I said, the reliable way to check PMI is through PMI. We are about to release mpich 4.1a1, which will have a separate
libpmi
package that third-party can directly use. A simple call toPMI_Init
should reveal enough information on the PMI versions and compatibility. We haven’t changed the PMI protocols ever, so it should work all the way back to old releases.PS: although making multiple
PMI_Init
for the purpose of version check is untested, but I think should work.