Unable to run TensorRTIS on AKS

I am trying to run TensortRTIS on AKS but get the following failure:

$ kubectl logs trt
trtserver: error while loading shared libraries: libnvidia-ml.so.1: cannot open shared object file: No such file or directory

I tried installing NVIDIA drivers, but it didnt help… (just running the simple-string example for CPU only… not running GPU) https://docs.microsoft.com/en-us/azure/aks/gpu-cluster

My podspec is:

apiVersion: v1
kind: Pod
metadata:
  name: trt
  namespace: default
spec:
  containers:
  - args:
    - --model-store=gs://rakelkar1/trt_sample
    - --allow-poll-model-repository=false
    - --allow-grpc=true
    - --allow-http=true
    - --grpc-port=9000
    - --rest-port=8080
    command:
    - trtserver
    env:
    - name: PORT
      value: "8080"
    image: nvcr.io/nvidia/tensorrtserver:19.05-py3
    imagePullPolicy: IfNotPresent
    name: user-container
    ports:
    - containerPort: 8080
      name: user-port
      protocol: TCP

[deadeyegoodwin]

TRTIS does support CPUs and should also work on servers that don’t have GPUs or CUDA installed. https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-master-branch-guide/docs/run.html#running-the-inference-server-on-a-system-without-a-gpu

When using kubernetes you need to use only “args” in your spec, like this: https://github.com/NVIDIA/tensorrt-inference-server/blob/master/deploy/single_server/templates/deployment.yaml#L59

Did you try the example Helm chart from the repo to see if that works for you? https://github.com/NVIDIA/tensorrt-inference-server/tree/master/deploy/single_server

[rakelkar]

works when trtserver is specified in args instead of command - thanks 😃