UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain

Description I upgraded the triton server with the ngc container 22.01-py3 and all my models becomes unavailable.

This previous issue suggests that the nvidia driver is outdated, so I did an upgrade and now I got: Driver Version: 495.46, CUDA Version: 11.5 on my host (ubuntu 20.04). But still doesn’t work.

Triton Information What version of Triton are you using? The one in docker image nvcr.io/nvidia/tritonserver:22.01-py3.

To Reproduce

docker run nvcr.io/nvidia/tritonserver:22.01-py3

Expected behavior It should not fail.

[TaylorHere]

I meet this issue when deploying tritonserver in kubernetes with A100 Card containerd://1.5.4 Driver Version: 470.82.01 CUDA Version: 11.6 nvcr tag: 22.03-py3

I was use tensorRT to build a model.plan, then run it with tritonserver, with the same nvcr container version tag but it says:

UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain

but I finally figure it out.

it can work with

command:
  - bash
args:
  - -c
  - tritonserver --model-store=/models/

can’t work with

command:
  - tritonserver
args:
  - --model-store=/models/

I have no idea why this happened.

[739982423]

@CoderHam Thanks for the pointers, I will give it a try for my local test environment. However, we would like to deploy to a K8s cluster, and according to the GPU Operator Component Matrix, the latest we can get is 470.82.01. Do you by chance know a workaround for this?

2022.2.14: Version 510.54 for RTX3090 is available for download at https://www.nvidia.com/Download/driverResults.aspx/186996/en-us