Python backend server stuck after initializing

Description Triton python backend server gets stuck after running the initialize function with no error message

Triton Information nvcr.io/nvidia/tritonserver:21.06-py3

Are you using the Triton container or did you build it yourself? Build it myself with the dockerfile below

FROM nvcr.io/nvidia/tritonserver:21.06-py3
RUN apt-get -y update
RUN apt-get install -y libgl1-mesa-glx

To Reproduce Details in the readme. Triton related files are all in the triton folder https://github.com/ernestlwt/ScaledYOLOv4

Expected behavior server unable to start. stuck after initialization with no error messages

I0802 07:53:57.695018 1 metrics.cc:291] Collecting metrics for GPU 0: GeForce RTX 2080 with Max-Q Design
I0802 07:53:57.943576 1 libtorch.cc:987] TRITONBACKEND_Initialize: pytorch
I0802 07:53:57.943615 1 libtorch.cc:997] Triton TRITONBACKEND API version: 1.4
I0802 07:53:57.943621 1 libtorch.cc:1003] 'pytorch' TRITONBACKEND API version: 1.4
2021-08-02 07:53:58.079715: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I0802 07:53:58.118937 1 tensorflow.cc:2165] TRITONBACKEND_Initialize: tensorflow
I0802 07:53:58.118959 1 tensorflow.cc:2175] Triton TRITONBACKEND API version: 1.4
I0802 07:53:58.118963 1 tensorflow.cc:2181] 'tensorflow' TRITONBACKEND API version: 1.4
I0802 07:53:58.118967 1 tensorflow.cc:2205] backend configuration:
{}
I0802 07:53:58.120069 1 onnxruntime.cc:1969] TRITONBACKEND_Initialize: onnxruntime
I0802 07:53:58.120082 1 onnxruntime.cc:1979] Triton TRITONBACKEND API version: 1.4
I0802 07:53:58.120086 1 onnxruntime.cc:1985] 'onnxruntime' TRITONBACKEND API version: 1.4
I0802 07:53:58.133451 1 openvino.cc:1188] TRITONBACKEND_Initialize: openvino
I0802 07:53:58.133469 1 openvino.cc:1198] Triton TRITONBACKEND API version: 1.4
I0802 07:53:58.133473 1 openvino.cc:1204] 'openvino' TRITONBACKEND API version: 1.4
I0802 07:53:58.251627 1 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f7dcc000000' with size 268435456
I0802 07:53:58.251971 1 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0802 07:53:58.253743 1 model_repository_manager.cc:1045] loading: yolov4:1
I0802 07:53:58.364911 1 python.cc:1267] Using Python execution env /yolov4_env.tar.gz
I0802 07:53:58.365099 1 python.cc:1489] TRITONBACKEND_ModelInstanceInitialize: yolov4_0 (CPU device 0)
W0802 07:53:59.697280 1 metrics.cc:396] Unable to get power limit for GPU 0: Success
W0802 07:54:01.700435 1 metrics.cc:396] Unable to get power limit for GPU 0: Success
W0802 07:54:03.702619 1 metrics.cc:396] Unable to get power limit for GPU 0: Success
Using CUDA device0 _CudaDeviceProperties(name='GeForce RTX 2080 with Max-Q Design', total_memory=7982MB)

Fusing layers... Model Summary: 331 layers, 7.07943e+07 parameters, 6.81919e+07 gradients
Initialized...

[ernestlwt]

Works fine with 21.07! thanks!

[dzier]

Triton uses a continuous deployment strategy with monthly releases, where we address bugs and issues in past versions by applying the fixes to the next version. Every release goes through a QA process to ensure we are releasing a stable version every release. However, bugs to occur and get through as users adopt Triton in new and interesting ways. We understand that some customers cannot always update Triton every month are working on a solution to address these customers. Until then, I would encourage Triton users to update to the latest release when possible. Please pay attention to the Known Issues section in the Release announcement for issues that may affect your usage of Triton.