Running SMARTS/ULTRA on Compute Canada

TODO: Introduction of this discussion

Feel free to put all questions, comments, etc. about running SMARTS and ULTRA on Compute Canada in this discussion issue.

Running SMARTS on Compute Canada

# Login to Compute Canada with Trusted X11 Forwarding and the forwarded port for Envision.
$ ssh <user-name>@<cluster-name>.computecanada.ca -Y -L localhost:8081:localhost:8081

# On your Compute Canada login node, obtain the Docker image for SMARTS and compress it (taken from
# https://docs.computecanada.ca/wiki/Singularity#Creating_an_image_using_Docker_Hub_and_Dockerfile).
$ cd ~/scratch
$ wget https://raw.githubusercontent.com/moby/moby/master/contrib/download-frozen-image-v2.sh
$ sh download-frozen-image-v2.sh smarts-0416_docker huaweinoah/smarts:v0.4.16
$ cd smarts-0416_docker && tar cf ../smarts-0416_docker.tar * && cd ..

# Start an interactive job and build the Singularity container.
$ cd ~/scratch
$ salloc --mem-per-cpu=2000 --cpus-per-task=4 --time=2:0:0 --x11
$ module load singularity
$ singularity build smarts-0416_singularity.sif docker-archive://smarts-0416_docker.tar
$ exit  # Exit out of the interactive job once the Singularity container is built.

# Move the Singularity container back to your projects directory and clone SMARTS.
$ cd ~/scratch
$ mv smarts-0416_singularity.sif ~/projects/<sponsor-name>/<user-name>/
$ cd ~/projects/<sponsor-name>/<user-name>/
$ git clone https://github.com/huawei-noah/SMARTS.git

# Execute the Singularity container and bind your SMARTS directory to the /SMARTS directory in the container.
# After, go to your the SMARTS directory in the container, modify the PYTHONPATH, and run an example!
$ cd ~/projects/<sponsor-name>/<user-name>/
$ singularity shell --bind SMARTS/:/SMARTS --env DISPLAY=$DISPLAY smarts-0416_singularity.sif
Singularity> cd /SMARTS
Singularity> export PYTHONPATH=/SMARTS:$PYTHONPATH
Singularity> supervisord

Running ULTRA on Compute Canada

Follow the steps above to obtain smarts-0416_singularity.sif and SMARTS/.

# Start an interactive job to run an ULTRA experiment.
$ salloc --time=1:0:0 --mem=16G --cpus-per-task=8 --ntasks=1
$ module load singularity
$ singularity shell --bind SMARTS/:/SMARTS --env DISPLAY=$DISPLAY smarts-0416_singularity.sif
Singularity> cd /SMARTS/ultra
Singularity> export PYTHONPATH=/SMARTS/ultra:/SMARTS/:$PYTHONPATH

# Follow instructions in https://github.com/huawei-noah/SMARTS/blob/master/ultra/docs/getting_started.md to
# run the experiment.

[christianjans]

Hi @mansur007, try binding just the certificate, something like:

singularity shell --bind /etc/pki/tls/certs/ca-bundle.crt,SMARTS:/SMARTS --env DISPLAY=$DISPLAY smarts-0416_singularity.sif

I will update the comment.

[christianjans]

@2017soft,

How can we install some additional packages in the container?

I believe if you also bind the certificates of the Compute Canada machine to the same directory of the Singularity container, then the certificates needed to install extra packages should be found. That is, run it like this:

$ singularity shell --bind /etc/pki/tls/certs/ca-bundle.crt,SMARTS/:/SMARTS smarts-0416_singularity.sif

And then try pip install ... or whatever.

This works for some packages, but I know when I tried to setup SMARTS this way, some packages wouldn’t install. If this is the case for you, perhaps I would try changing the Dockerfile of SMARTS so that it also installs the packages you need. https://github.com/huawei-noah/SMARTS/blob/c5a16088152e840dbaa9afe302e8bb9338bc6a93/Dockerfile#L62-L63 Then you can build and push this new image to your DockerHub (see http://jsta.github.io/r-docker-tutorial/04-Dockerhub.html), and then use this image instead of huaweinoah/smarts:v0.4.16.