NVIDIA drivers not installing on Azure cloud runner

Hi everybody I am trying to use cml-runner on GitLab to deploy a GPU machine on which to run training. The deployment works great but the docker container then running the training can’t find any NVIDIA drivers it seems, as I can’t run ‘nvidia-smi’.

My .gitlab-ci.yml looks like this (simplified):

stages:
  - deploy
  - train

deploy:
  stage: deploy
  when: always
  image: dvcorg/cml:0-dvc2-base1-gpu
  script:
    - cml-runner
      --cloud azure
      --cloud-region eu-west
      --cloud-type Standard_NC4as_T4_v3
      --cloud-hdd-size 128
      --cloud-gpu v100
      --labels=cml-runner-gpu

train:
  stage: train
  when: on_success
  image: dvcorg/cml:0-dvc2-base1-gpu
  tags:
    - cml-runner-gpu

  script:
    - nvidia-smi

In the examples it’s not mentioned that I need to install the drivers myself on the deployed machine, it looks like it should work out-of-the-box, or am I overlooking something? Is that only for AWS? Do I need to pass a script installing the drivers through --cloud-startup-script?

Cheers

[MaxHuerlimann]

Hi, the issue is still persisting. If I pass a startup script that installs the drivers it works, though, so like this:

stages:
  - deploy
  - train

deploy:
  stage: deploy
  when: always
  image: dvcorg/cml:0-dvc2-base1-gpu
  script:
    - script=$(echo 'sudo apt-get update && sudo apt-get upgrade && sudo apt-get install -y nvidia-driver-460' | base64 --wrap 0)
    - cml-runner
      --cloud azure
      --cloud-region eu-west
      --cloud-type Standard_NC4as_T4_v3
      --cloud-hdd-size 128
      --cloud-startup-script $script
      --labels=cml-runner-gpu

train:
  stage: train
  when: on_success
  image: dvcorg/cml:0-dvc2-base1-gpu
  tags:
    - cml-runner-gpu

  script:
    - nvidia-smi

I even manually ssh’d to the created machine and could confirm that no NVIDIA drivers were installed if I didn’t pass the above script.

[MaxHuerlimann]

That’s great! Thanks for the quick fix 😃