Running through Dockerfile broken

Describe the bug When using an image based on the provided Dockerfile and running the quick start steps (download enron data, run deep.py), execution crashes before training begins.

To Reproduce Steps to reproduce the behavior:

1. Build an image using the provided Dockerfile
2. Run said image, mounting 8 RTX800 GPUs
3. Fetch enron data using the prepare_dataset.py script
4. Run ./deepy.py pretrain_gpt2.py -d configs small.yml local_configs.yml
5. The code crashes with a non-descript NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled system error, NCCL version 2.7.8

Expected behavior Training starts or a specific error is provided.

Proposed solution The NCCL error is typically a stand-in for a real issue that is not relayed back through multiprocessing. As a first step, it would be nice to know if this setup works out-of-the-box for others; in that case, it might be my resources or CUDA version.

Environment (please complete the following information):

• GPUs: 8 RTX8000 GPUs
• Configs: Ubuntu 20.04, Cuda 11.2

Additional context Add any other context about the problem here.

[VHellendoorn]

That worked! Adding –shm-size=1g –ulimit memlock=-1 to the nvidia-docker run (for others: note that this won’t work with docker run --runtime=nvidia) command solved it. The default shared memory was 64MB, which is evidently far too little.

Thanks so much for your help!

[PyxAI]

That worked! Adding –shm-size=1g –ulimit memlock=-1 to the nvidia-docker run (for others: note that this won’t work with docker run --runtime=nvidia) command solved it. The default shared memory was 64MB, which is evidently far too little.

Thanks so much for your help!

That worked for me too Just want to mention that the dashes in the command here needs to be replaced -shm-size=1g -ulimit memlock=-1 – != -