Server start stuck when loading python model instantiating certain transformers model

Description When startimg triton server with a python model which instantiates certain hugging face transformers models during initialize() the server gets stuck. Namely I tried it with this checkpoint. Since it works if I use this checkpoint instead, I guess the Problem could be the model size? I also experienced a similiar Problem when using the aforementioned working checkpoint but also loading other models (onnx). Could this be related to memory/gpu settings of the docker container?

Triton Information nvcr.io/nvidia/tritonserver:22.04-py3

To Reproduce env:

conda create --name triton-transformers python=3.8.10
conda activate triton-transformers
conda install -c huggingface transformers==4.14.1 tokenizers==0.10.3
export PYTHONNOUSERSITE=True
comda-pack

model:

...
from transformers import AutoModelForTokenClassification,

class TritonPythonModel:
    def initialize(self, args):
        self.checkpoint = r"/checkpoints/tner-xlm-roberta-large-multiconer-multi/"
        self.ner_model = AutoModelForTokenClassification.from_pretrained(self.checkpoint)
        print('Initialized...')

    def execute(self, requests):
        ...

config:

backend: "python"
max_batch_size: 8
input [
  {
    name: "sequence"
    data_type: TYPE_STRING
    dims: [ -1, -1 ]
  }
]
output [
  {
    name: "entities"
    data_type: TYPE_STRING
    dims: [ -1, -1 ]
  }
]
parameters: [
  {
    key: "EXECUTION_ENV_PATH"
    value: { string_value: "/envs/triton-transformers.tar.gz"}
  }
]

Expected behavior Triton Server should just start.

[AJHoeh]

Thanks at both of you!

[dyastremsky]

Thanks for the detailed post and clear reproduction instructions. It’s possible and a quick test of that would be increasing the GPUs/memory available to the container, if possible. To help find the cause more quickly, would you be able to run it with the --log-verbose=1 flag and provide the verbose logs?