[Custom python environment]

Hi all !

In the context of cluster computing, it is sometime necessary to have the jobs running with a local python environment and not the one from the central one (where the job has been launched).

Currently, the python executable path is automatically extracted with shlex.quote(sys.executable) in slurm.py before being used to generate the scratch file in _make_sbatch_string.

While this is a great default behaviour, we would argue with @MJHutchinson to be able to specify a custom python executable path to override the default one, as implemented in this commit. The key line is:

 f"srun --output {stdout} --error {stderr} --unbuffered {executable} {command}",

Then, it combination with hydra, it is sufficient to add executable:/data/localhost/not-backed-up/${env:USER}/utils/venv_projec_name/bin/python in the slurm config.

If that sounds like a good solution to you all, I’ll push a PR, otherwise I’m happy to discuss the issue.

Additionally, it seems that it is now possible to execute commands before running srun cf hydra_submitit_launcher. Would be great to also have a teardown argument to execute commands after running srun.

Best, Emile

[cgerum]

One additional problem is then running with singularity where executable should usually be something along the lines of:

singularity --nv exec my_image.sif /opt/python3.8/bin/python

Which will break when running shlex.quote() on the executable string. Singularity is the preferred way of building environments on all SLURM cluster i have worked with.

[tileb1]

Hi @gwenzek, I also tried the branch and it works with my singularity image. It would be great to have it merged!