Docker install not finding GPUs

On a p3.2xlarge, the GPU Docker install (built with the --gpu flag) is not finding GPU devices. From ts_log.log - note the “Number of GPUs” line:

model-server@a8819511cfaa:~/logs$ more ts_log.log
2020-04-16 01:53:17,099 [INFO ] main org.pytorch.serve.ModelServer -
TS Home: /usr/local/lib/python3.6/dist-packages
Current directory: /home/model-server
Temp directory: /home/model-server/tmp
Number of GPUs: 0
Number of CPUs: 8
Max heap size: 13644 M
Python executable: /usr/bin/python3
Config file: /home/model-server/config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Model Store: /home/model-server/model-store
Initial Models: N/A
Log dir: /home/model-server/logs
Metrics dir: /home/model-server/logs
Netty threads: 32
Netty client threads: 0
Default workers per model: 8
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500

When starting on a GPU machine with CUDA drivers installed, I normally see Number of GPUs: 4.

[fbbradheintz]

That didn’t work either. On a p3.8xlarge (which has 4 CUDA-compatible GPUs), I built the Docker image with the --gpu directive, as the instructions prescribed. I then issued the same command line @harshbafna listed above, and got this result:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: cuda error: no cuda-capable device is detected\\\\n\\\"\"": unknown.

This is on DL AMI 27, Ubuntu 18, current master branch.

[chauhang]

@maaquib Please see the comments left in the commit and address them.