Error: Failed preparing Gitlab runner: Forbidden in cml-cloud-case

I have updated the images in the yaml to discover that the repo is failing . My personal fashion_mnist uses the images and its OK. The only difference is the machine size.

{"level":"info","message":"Preparing workdir /home/runner..."}
{"level":"info","message":"Deploying cloud runner plan..."}
{"level":"info","message":"Terraform apply..."}
{"level":"error","message":"terraform -chdir='/home/runner' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  \u001b[32m+\u001b[0m create\n\u001b[0m\nTerraform will perform the following actions:\n\n\u001b[1m  # iterative_cml_runner.runner\u001b[0m will be created\u001b[0m\u001b[0m\n\u001b[0m  \u001b[32m+\u001b[0m\u001b[0m resource \"iterative_cml_runner\" \"runner\" {\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mcloud\u001b[0m\u001b[0m                = \"aws\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mdriver\u001b[0m\u001b[0m               = \"gitlab\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mid\u001b[0m\u001b[0m                   = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0midle_timeout\u001b[0m\u001b[0m         = 300\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mimage\u001b[0m\u001b[0m                = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_hdd_size\u001b[0m\u001b[0m    = 64\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_ip\u001b[0m\u001b[0m          = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_launch_time\u001b[0m\u001b[0m = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_type\u001b[0m\u001b[0m        = \"g3.4xlarge\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mlabels\u001b[0m\u001b[0m               = \"cml-runner-gpu\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mname\u001b[0m\u001b[0m                 = \"cml-gp06y1oau8\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mregion\u001b[0m\u001b[0m               = \"us-west\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mrepo\u001b[0m\u001b[0m                 = \"https://gitlab.com/iterative.ai/cml-cloud-case\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0msingle\u001b[0m\u001b[0m               = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot\u001b[0m\u001b[0m                 = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot_price\u001b[0m\u001b[0m           = -1\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mssh_public\u001b[0m\u001b[0m           = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mtoken\u001b[0m\u001b[0m                = (sensitive value)\n    }\n\n\u001b[0m\u001b[1mPlan:\u001b[0m 1 to add, 0 to change, 0 to destroy.\n\u001b[0m\u001b[0m\u001b[1miterative_cml_runner.runner: Creating...\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [30s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [40s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [50s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m0s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m30s elapsed]\u001b[0m\u001b[0m\n\n\t\u001b[31m╷\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mError checking the runner status\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[0m  with iterative_cml_runner.runner,\n\u001b[31m│\u001b[0m \u001b[0m  on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n\u001b[31m│\u001b[0m \u001b[0m  14: resource \"iterative_cml_runner\" \"runner\" \u001b[4m{\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m-- Logs begin at Thu 2021-09-30 13:32:55 UTC, end at Mon 2021-12-06\n\u001b[31m│\u001b[0m \u001b[0m12:33:58 UTC. --\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:35 ip-172-31-2-224 systemd[1]: Started cml.service.\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Preparing workdir\n\u001b[31m│\u001b[0m \u001b[0m/tmp/tmp.PdTeIq49Jr/.cml/cml-gp06y1oau8...\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Launching gitlab runner\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"error\",\"message\":\"Failed preparing Gitlab runner:\n\u001b[31m│\u001b[0m \u001b[0mForbidden\",\"stack\":\"Error: Failed preparing Gitlab runner: Forbidden\\n\n\u001b[31m│\u001b[0m \u001b[0mat Gitlab.startRunner\n\u001b[31m│\u001b[0m \u001b[0m(/usr/lib/node_modules/@dvcorg/cml/src/drivers/gitlab.js:209:13)\\n    at\n\u001b[31m│\u001b[0m \u001b[0mprocessTicksAndRejections\n\u001b[31m│\u001b[0m \u001b[0m(internal/process/task_queues.js:97:5)\",\"status\":\"terminated\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"waiting 20 seconds before exiting...\"}\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m╵\u001b[0m\u001b[0m\n","stack":"Error: terraform -chdir='/home/runner' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  \u001b[32m+\u001b[0m create\n\u001b[0m\nTerraform will perform the following actions:\n\n\u001b[1m  # iterative_cml_runner.runner\u001b[0m will be created\u001b[0m\u001b[0m\n\u001b[0m  \u001b[32m+\u001b[0m\u001b[0m resource \"iterative_cml_runner\" \"runner\" {\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mcloud\u001b[0m\u001b[0m                = \"aws\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mdriver\u001b[0m\u001b[0m               = \"gitlab\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mid\u001b[0m\u001b[0m                   = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0midle_timeout\u001b[0m\u001b[0m         = 300\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mimage\u001b[0m\u001b[0m                = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_hdd_size\u001b[0m\u001b[0m    = 64\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_ip\u001b[0m\u001b[0m          = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_launch_time\u001b[0m\u001b[0m = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_type\u001b[0m\u001b[0m        = \"g3.4xlarge\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mlabels\u001b[0m\u001b[0m               = \"cml-runner-gpu\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mname\u001b[0m\u001b[0m                 = \"cml-gp06y1oau8\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mregion\u001b[0m\u001b[0m               = \"us-west\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mrepo\u001b[0m\u001b[0m                 = \"https://gitlab.com/iterative.ai/cml-cloud-case\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0msingle\u001b[0m\u001b[0m               = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot\u001b[0m\u001b[0m                 = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot_price\u001b[0m\u001b[0m           = -1\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mssh_public\u001b[0m\u001b[0m           = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mtoken\u001b[0m\u001b[0m                = (sensitive value)\n    }\n\n\u001b[0m\u001b[1mPlan:\u001b[0m 1 to add, 0 to change, 0 to destroy.\n\u001b[0m\u001b[0m\u001b[1miterative_cml_runner.runner: Creating...\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [30s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [40s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [50s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m0s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m30s elapsed]\u001b[0m\u001b[0m\n\n\t\u001b[31m╷\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mError checking the runner status\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[0m  with iterative_cml_runner.runner,\n\u001b[31m│\u001b[0m \u001b[0m  on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n\u001b[31m│\u001b[0m \u001b[0m  14: resource \"iterative_cml_runner\" \"runner\" \u001b[4m{\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m-- Logs begin at Thu 2021-09-30 13:32:55 UTC, end at Mon 2021-12-06\n\u001b[31m│\u001b[0m \u001b[0m12:33:58 UTC. --\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:35 ip-172-31-2-224 systemd[1]: Started cml.service.\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Preparing workdir\n\u001b[31m│\u001b[0m \u001b[0m/tmp/tmp.PdTeIq49Jr/.cml/cml-gp06y1oau8...\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Launching gitlab runner\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"error\",\"message\":\"Failed preparing Gitlab runner:\n\u001b[31m│\u001b[0m \u001b[0mForbidden\",\"stack\":\"Error: Failed preparing Gitlab runner: Forbidden\\n\n\u001b[31m│\u001b[0m \u001b[0mat Gitlab.startRunner\n\u001b[31m│\u001b[0m \u001b[0m(/usr/lib/node_modules/@dvcorg/cml/src/drivers/gitlab.js:209:13)\\n    at\n\u001b[31m│\u001b[0m \u001b[0mprocessTicksAndRejections\n\u001b[31m│\u001b[0m \u001b[0m(internal/process/task_queues.js:97:5)\",\"status\":\"terminated\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"waiting 20 seconds before exiting...\"}\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m╵\u001b[0m\u001b[0m\n\n    at /usr/lib/node_modules/@dvcorg/cml/src/utils.js:14:27\n    at ChildProcess.exithandler (node:child_process:404:5)\n    at ChildProcess.emit (node:events:390:28)\n    at maybeClose (node:internal/child_process:1064:16)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)","status":"terminated"}
{"level":"info","message":"waiting 20 seconds before exiting..."}

[casperdcl]

The Error: Failed preparing Gitlab runner: Forbidden is rather hidden. Wonder if we can improve the messaging a bit 😃

[0x2b3bfa0]

Failed preparing Gitlab runner: Forbidden

Expired token or something?