Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

python backend complains about modulenotfound

See original GitHub issue

Description import librosa in model.py raises ModuleNotFoundError: No module name d 'decorator'

Triton Information

nvcr.io/nvidia/tritonserver:21.07-py3

To Reproduce

Create conda environment


$  conda create -y -n test2 python=3.8
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/user/miniconda3/envs/test2

  added / updated specs:
    - python=3.8


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python-3.8.11              |h12debd9_0_cpython        18.2 MB
    ------------------------------------------------------------
                                           Total:        18.2 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.7.5-h06a4308_1
  certifi            pkgs/main/linux-64::certifi-2021.5.30-py38h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1k-h27cfd23_0
  pip                pkgs/main/linux-64::pip-21.1.3-py38h06a4308_0
  python             pkgs/main/linux-64::python-3.8.11-h12debd9_0_cpython
  readline           pkgs/main/linux-64::readline-8.1-h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-52.0.0-py38h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.36.0-hc218d9a_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  wheel              pkgs/main/noarch::wheel-0.36.2-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3



Downloading and Extracting Packages
python-3.8.11        | 18.2 MB   | ################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test2
#
# To deactivate an active environment, use
#
#     $ conda deactivate


$ conda activate test2

$  pip install conda-pack librosa
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting conda-pack
  Downloading conda-pack-0.6.0.tar.gz (43 kB)
     |ââââââââââââââââââââââââââââââââ| 43 kB 1.8 MB/s
Collecting librosa
  Downloading librosa-0.8.1-py3-none-any.whl (203 kB)
     |ââââââââââââââââââââââââââââââââ| 203 kB 3.1 MB/s
Requirement already satisfied: setuptools in /home/andy/workspace/miniconda3/envs/test2/lib/python3.8/site-packages (from conda-pack) (52.0.0.post20210125)
Collecting scikit-learn!=0.19.0,>=0.14.0
  Downloading scikit_learn-0.24.2-cp38-cp38-manylinux2010_x86_64.whl (24.9 MB)
     |ââââââââââââââââââââââââââââââââ| 24.9 MB 22.4 MB/s
Collecting joblib>=0.14
  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)
     |ââââââââââââââââââââââââââââââââ| 303 kB 80.6 MB/s
Collecting numba>=0.43.0
  Downloading numba-0.53.1-cp38-cp38-manylinux2014_x86_64.whl (3.4 MB)
     |ââââââââââââââââââââââââââââââââ| 3.4 MB 77.5 MB/s
Collecting resampy>=0.2.2
  Downloading resampy-0.2.2.tar.gz (323 kB)
     |ââââââââââââââââââââââââââââââââ| 323 kB 78.1 MB/s
Collecting packaging>=20.0
  Downloading packaging-21.0-py3-none-any.whl (40 kB)
     |ââââââââââââââââââââââââââââââââ| 40 kB 54.0 MB/s
Collecting scipy>=1.0.0
  Downloading scipy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.4 MB)
     |ââââââââââââââââââââââââââââââââ| 28.4 MB 95.6 MB/s
Collecting numpy>=1.15.0
  Downloading numpy-1.21.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.8 MB)
     |ââââââââââââââââââââââââââââââââ| 15.8 MB 93.8 MB/s
Collecting pooch>=1.0
  Downloading pooch-1.4.0-py3-none-any.whl (51 kB)
     |ââââââââââââââââââââââââââââââââ| 51 kB 24.1 MB/s
Requirement already satisfied: decorator>=3.0.0 in /home/andy/.local/lib/python3.8/site-packages (from librosa) (5.0.7)
Collecting soundfile>=0.10.2
  Downloading SoundFile-0.10.3.post1-py2.py3-none-any.whl (21 kB)
Collecting audioread>=2.0.0
  Downloading audioread-2.1.9.tar.gz (377 kB)
     |ââââââââââââââââââââââââââââââââ| 377 kB 86.6 MB/s
Collecting llvmlite<0.37,>=0.36.0rc1
  Downloading llvmlite-0.36.0-cp38-cp38-manylinux2010_x86_64.whl (25.3 MB)
     |ââââââââââââââââââââââââââââââââ| 25.3 MB 89.6 MB/s
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |ââââââââââââââââââââââââââââââââ| 67 kB 60.7 MB/s
Collecting appdirs
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting requests
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |ââââââââââââââââââââââââââââââââ| 62 kB 35.8 MB/s
Requirement already satisfied: six>=1.3 in /home/andy/.local/lib/python3.8/site-packages (from resampy>=0.2.2->librosa) (1.16.0)
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-2.2.0-py3-none-any.whl (12 kB)
Collecting cffi>=1.0
  Downloading cffi-1.14.6-cp38-cp38-manylinux1_x86_64.whl (411 kB)
     |ââââââââââââââââââââââââââââââââ| 411 kB 89.6 MB/s
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |ââââââââââââââââââââââââââââââââ| 112 kB 89.8 MB/s
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
     |ââââââââââââââââââââââââââââââââ| 138 kB 89.7 MB/s
Requirement already satisfied: certifi>=2017.4.17 in /home/andy/workspace/miniconda3/envs/test2/lib/python3.8/site-packages (from requests->pooch>=1.0->librosa) (2021.5.30)
Collecting idna<4,>=2.5
  Downloading idna-3.2-py3-none-any.whl (59 kB)
     |ââââââââââââââââââââââââââââââââ| 59 kB 75.7 MB/s
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Building wheels for collected packages: conda-pack, audioread, resampy
  Building wheel for conda-pack (setup.py) ... done
  Created wheel for conda-pack: filename=conda_pack-0.6.0-py2.py3-none-any.whl size=30893 sha256=3a483159f1e919ff3c1104b411958c3a3b66c8c0e9037bc72ebff9d180712a22
  Stored in directory: /tmp/pip-ephem-wheel-cache-1o200pri/wheels/56/1b/9e/0da27a4c18349d8f048a8fe87d763d75d3098384e9fa285e45
  Building wheel for audioread (setup.py) ... done
  Created wheel for audioread: filename=audioread-2.1.9-py3-none-any.whl size=23141 sha256=c3eefc897b7d45a17d4c07404e4a26a269b45efb8b7b162f78bb16e543c08877
  Stored in directory: /tmp/pip-ephem-wheel-cache-1o200pri/wheels/49/5a/e4/df590783499a992a88de6c0898991d1167453a3196d0d1eeb7
  Building wheel for resampy (setup.py) ... done
  Created wheel for resampy: filename=resampy-0.2.2-py3-none-any.whl size=320718 sha256=bbfb6df8f7e542e9637f85fbe0452da1ca2ae08a004d4b6f7b8839bbaa2e8f29
  Stored in directory: /tmp/pip-ephem-wheel-cache-1o200pri/wheels/6f/d1/5d/f13da53b1dcbc2624ff548456c9ffb526c914f53c12c318bb4
Successfully built conda-pack audioread resampy
Installing collected packages: urllib3, pyparsing, pycparser, numpy, llvmlite, idna, charset-normalizer, threadpoolctl, scipy, requests, packaging, numba, joblib, cffi, appdirs, soundfile, scikit-learn, resampy, pooch, audioread, librosa, conda-pack
Successfully installed appdirs-1.4.4 audioread-2.1.9 cffi-1.14.6 charset-normalizer-2.0.4 conda-pack-0.6.0 idna-3.2 joblib-1.0.1 librosa-0.8.1 llvmlite-0.36.0 numba-0.53.1 numpy-1.21.1 packaging-21.0 pooch-1.4.0 pycparser-2.20 pyparsing-2.4.7 requests-2.26.0 resampy-0.2.2 scikit-learn-0.24.2 scipy-1.7.1 soundfile-0.10.3.post1 threadpoolctl-2.2.0 urllib3-1.26.6

$ python
Python 3.8.11 (default, Aug  3 2021, 15:09:35) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import librosa
>>> librosa.__version__
'0.8.1'
>>> import decorator
>>> decorator.__version__
'5.0.7'
>>> 

$ conda-pack
Collecting packages...
Packing environment at '/home/user/miniconda3/envs/test2' to 'test2.tar.gz'
[########################################] | 100% Completed | 15.7s

model.py

import json
import os
import librosa
import triton_python_backend_utils as pb_utils


class TritonPythonModel:

    def initialize(self, args):
        print("Init")

    def execute(self, requests):
        print("Execute")
        return []

    def finalize(self):
        print('Cleaning up...')

config.pbtxt

backend: "python"

input [
  {
    name: "INPUT0"
    data_type: TYPE_STRING
    dims: [ -1 ]
  },
  {
    name: "INPUT1"
    data_type: TYPE_STRING
    dims: [ -1 ]
  }
]

output [
  {
    name: "OUTPUT0"
    data_type: TYPE_STRING
    dims: [ -1 ]
  }
]

max_batch_size: 1

instance_group [{ count: 1, kind: KIND_GPU }]

version_policy: { latest { num_versions : 1 }}

parameters: {
  key: "EXECUTION_ENV_PATH",
  value: {string_value: "/models/wav2lip/test2.tar.gz"}
}

Output

$ docker run --gpus=1 --shm-size=5G -p8000:8000 -p8001:8001 -p8002:8002 --rm -v /path/to/model_repository:/models --name tritonserver nvcr.io/nvidia/tritonserver:21.07-py3 tritonserver --model-repository=/models --log-verbose 20

=============================
== Triton Inference Server ==
=============================

NVIDIA Release 21.07 (build 24810355)

Copyright (c) 2018-2021, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

I0803 16:17:14.844166 1 metrics.cc:290] Collecting metrics for GPU 0: Tesla V100-PCIE-16GB
I0803 16:17:14.844530 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/pytorch/libtriton_pytorch.so
I0803 16:17:15.143440 1 libtorch.cc:998] TRITONBACKEND_Initialize: pytorch
I0803 16:17:15.143482 1 libtorch.cc:1008] Triton TRITONBACKEND API version: 1.4
I0803 16:17:15.143498 1 libtorch.cc:1014] 'pytorch' TRITONBACKEND API version: 1.4
I0803 16:17:15.143561 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/tensorflow1/libtriton_tensorflow1.so
2021-08-03 16:17:15.297360: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I0803 16:17:15.340564 1 tensorflow.cc:2169] TRITONBACKEND_Initialize: tensorflow
I0803 16:17:15.340600 1 tensorflow.cc:2179] Triton TRITONBACKEND API version: 1.4
I0803 16:17:15.340606 1 tensorflow.cc:2185] 'tensorflow' TRITONBACKEND API version: 1.4
I0803 16:17:15.340611 1 tensorflow.cc:2209] backend configuration:
{}
I0803 16:17:15.340660 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so
I0803 16:17:15.341998 1 onnxruntime.cc:1970] TRITONBACKEND_Initialize: onnxruntime
I0803 16:17:15.342022 1 onnxruntime.cc:1980] Triton TRITONBACKEND API version: 1.4
I0803 16:17:15.342028 1 onnxruntime.cc:1986] 'onnxruntime' TRITONBACKEND API version: 1.4
I0803 16:17:15.351287 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/openvino/libtriton_openvino.so
I0803 16:17:15.363374 1 openvino.cc:1193] TRITONBACKEND_Initialize: openvino
I0803 16:17:15.363400 1 openvino.cc:1203] Triton TRITONBACKEND API version: 1.4
I0803 16:17:15.363406 1 openvino.cc:1209] 'openvino' TRITONBACKEND API version: 1.4
I0803 16:17:15.751780 1 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f97f8000000' with size 268435456
I0803 16:17:15.752149 1 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0803 16:17:15.752725 1 backend_factory.h:45] Create TritonBackendFactory
I0803 16:17:15.752761 1 plan_backend_factory.cc:49] Create PlanBackendFactory
I0803 16:17:15.752767 1 plan_backend_factory.cc:56] Registering TensorRT Plugins
I0803 16:17:15.752814 1 logging.cc:52] Registered plugin creator - ::GridAnchor_TRT version 1
I0803 16:17:15.752833 1 logging.cc:52] Registered plugin creator - ::GridAnchorRect_TRT version 1
I0803 16:17:15.752853 1 logging.cc:52] Registered plugin creator - ::NMS_TRT version 1
I0803 16:17:15.752867 1 logging.cc:52] Registered plugin creator - ::Reorg_TRT version 1
I0803 16:17:15.752877 1 logging.cc:52] Registered plugin creator - ::Region_TRT version 1
I0803 16:17:15.752896 1 logging.cc:52] Registered plugin creator - ::Clip_TRT version 1
I0803 16:17:15.752911 1 logging.cc:52] Registered plugin creator - ::LReLU_TRT version 1
I0803 16:17:15.752920 1 logging.cc:52] Registered plugin creator - ::PriorBox_TRT version 1
I0803 16:17:15.752930 1 logging.cc:52] Registered plugin creator - ::Normalize_TRT version 1
I0803 16:17:15.752939 1 logging.cc:52] Registered plugin creator - ::ScatterND version 1
I0803 16:17:15.752951 1 logging.cc:52] Registered plugin creator - ::RPROI_TRT version 1
I0803 16:17:15.752962 1 logging.cc:52] Registered plugin creator - ::BatchedNMS_TRT version 1
I0803 16:17:15.752972 1 logging.cc:52] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
I0803 16:17:15.752980 1 logging.cc:52] Registered plugin creator - ::FlattenConcat_TRT version 1
I0803 16:17:15.752989 1 logging.cc:52] Registered plugin creator - ::CropAndResize version 1
I0803 16:17:15.753000 1 logging.cc:52] Registered plugin creator - ::DetectionLayer_TRT version 1
I0803 16:17:15.753009 1 logging.cc:52] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
I0803 16:17:15.753017 1 logging.cc:52] Registered plugin creator - ::EfficientNMS_TRT version 1
I0803 16:17:15.753027 1 logging.cc:52] Registered plugin creator - ::Proposal version 1
I0803 16:17:15.753041 1 logging.cc:52] Registered plugin creator - ::ProposalLayer_TRT version 1
I0803 16:17:15.753049 1 logging.cc:52] Registered plugin creator - ::PyramidROIAlign_TRT version 1
I0803 16:17:15.753057 1 logging.cc:52] Registered plugin creator - ::ResizeNearest_TRT version 1
I0803 16:17:15.753072 1 logging.cc:52] Registered plugin creator - ::Split version 1
I0803 16:17:15.753080 1 logging.cc:52] Registered plugin creator - ::SpecialSlice_TRT version 1
I0803 16:17:15.753098 1 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 1
I0803 16:17:15.753109 1 ensemble_backend_factory.cc:47] Create EnsembleBackendFactory
I0803 16:17:15.754510 1 model_repository_manager.cc:749] AsyncLoad() 'wav2lip'
I0803 16:17:15.754572 1 model_repository_manager.cc:988] TriggerNextAction() 'wav2lip' version 1: 1
I0803 16:17:15.754586 1 model_repository_manager.cc:1026] Load() 'wav2lip' version 1
I0803 16:17:15.754591 1 model_repository_manager.cc:1045] loading: wav2lip:1
I0803 16:17:15.854861 1 model_repository_manager.cc:1105] CreateInferenceBackend() 'wav2lip' version 1
I0803 16:17:15.854994 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/python/libtriton_python.so
I0803 16:17:15.857100 1 python.cc:1306] 'python' TRITONBACKEND API version: 1.4
I0803 16:17:15.857119 1 python.cc:1328] backend configuration:
{}
I0803 16:17:15.857131 1 python.cc:1405] shm-default-byte-size=67108864,shm-growth-byte-size=67108864,stub-timeout-seconds=30
I0803 16:17:15.857491 1 python.cc:1453] TRITONBACKEND_ModelInitialize: wav2lip (version 1)
I0803 16:17:15.858434 1 model_config_utils.cc:1521] ModelConfig 64-bit fields:
I0803 16:17:15.858451 1 model_config_utils.cc:1523]     ModelConfig::dynamic_batching::default_queue_policy::default_timeout_microseconds
I0803 16:17:15.858457 1 model_config_utils.cc:1523]     ModelConfig::dynamic_batching::max_queue_delay_microseconds
I0803 16:17:15.858461 1 model_config_utils.cc:1523]     ModelConfig::dynamic_batching::priority_queue_policy::value::default_timeout_microseconds
I0803 16:17:15.858465 1 model_config_utils.cc:1523]     ModelConfig::ensemble_scheduling::step::model_version
I0803 16:17:15.858470 1 model_config_utils.cc:1523]     ModelConfig::input::dims
I0803 16:17:15.858474 1 model_config_utils.cc:1523]     ModelConfig::input::reshape::shape
I0803 16:17:15.858479 1 model_config_utils.cc:1523]     ModelConfig::instance_group::secondary_devices::device_id
I0803 16:17:15.858483 1 model_config_utils.cc:1523]     ModelConfig::model_warmup::inputs::value::dims
I0803 16:17:15.858487 1 model_config_utils.cc:1523]     ModelConfig::optimization::cuda::graph_spec::graph_lower_bound::input::value::dim
I0803 16:17:15.858492 1 model_config_utils.cc:1523]     ModelConfig::optimization::cuda::graph_spec::input::value::dim
I0803 16:17:15.858497 1 model_config_utils.cc:1523]     ModelConfig::output::dims
I0803 16:17:15.858501 1 model_config_utils.cc:1523]     ModelConfig::output::reshape::shape
I0803 16:17:15.858505 1 model_config_utils.cc:1523]     ModelConfig::sequence_batching::direct::max_queue_delay_microseconds
I0803 16:17:15.858510 1 model_config_utils.cc:1523]     ModelConfig::sequence_batching::max_sequence_idle_microseconds
I0803 16:17:15.858514 1 model_config_utils.cc:1523]     ModelConfig::sequence_batching::oldest::max_queue_delay_microseconds
I0803 16:17:15.858519 1 model_config_utils.cc:1523]     ModelConfig::version_policy::specific::versions
I0803 16:17:15.858619 1 python.cc:1275] Using Python execution env /models/wav2lip/test2.tar.gz
I0803 16:17:15.859957 1 python.cc:1497] TRITONBACKEND_ModelInstanceInitialize: wav2lip_0 (GPU device 0)
I0803 16:17:15.861238 1 backend_model_instance.cc:105] Creating instance wav2lip_0 on GPU 0 (7.0) using artifact ''
I0803 16:17:23.186869 56 python.cc:918] Starting Python backend stub: export LD_LIBRARY_PATH=/tmp/python_env_z0IKPf/0/lib:$LD_LIBRARY_PATH; source /tmp/python_env_z0IKPf/0/bin/activate && exec /opt/tritonserver/backends/python/triton_python_backend_stub /models/wav2lip/1/model.py /wav2lip_0_GPU_0 67108864 67108864 1 /opt/tritonserver/backends/python
0803 16:17:23.796501 56 pb_stub.cc:639] ModuleNotFoundError: No module named 'decorator'

At:
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/_cache.py(40): wrapper
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/util/utils.py(230): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/util/__init__.py(83): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(961): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/notation.py(8): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(1050): _handle_fromlist
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/convert.py(7): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/__init__.py(5): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(1050): _handle_fromlist
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/__init__.py(211): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /models/wav2lip/1/model.py(13): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load

I0803 16:17:23.796665 1 python.cc:1557] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0803 16:17:23.796922 1 python.cc:1476] TRITONBACKEND_ModelFinalize: delete model state
I0803 16:17:23.796953 1 triton_backend_manager.cc:101] unloading backend 'python'
I0803 16:17:23.796963 1 python.cc:1433] TRITONBACKEND_Finalize: Start
I0803 16:17:24.138899 1 python.cc:1438] TRITONBACKEND_Finalize: End
E0803 16:17:24.141075 1 model_repository_manager.cc:1215] failed to load 'wav2lip' version 1: Internal: ModuleNotFoundError: No module named 'decorator'

At:
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/_cache.py(40): wrapper
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/util/utils.py(230): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/util/__init__.py(83): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(961): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/notation.py(8): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(1050): _handle_fromlist
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/convert.py(7): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/__init__.py(5): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(1050): _handle_fromlist
  /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/__init__.py(211): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load
  /models/wav2lip/1/model.py(13): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(843): exec_module
  <frozen importlib._bootstrap>(686): _load_unlocked
  <frozen importlib._bootstrap>(975): _find_and_load_unlocked
  <frozen importlib._bootstrap>(991): _find_and_load

I0803 16:17:24.141103 1 model_repository_manager.cc:988] TriggerNextAction() 'wav2lip' version 1: 0
I0803 16:17:24.141112 1 model_repository_manager.cc:1003] no next action, trigger OnComplete()
I0803 16:17:24.141197 1 model_repository_manager.cc:594] VersionStates() 'wav2lip'
I0803 16:17:24.141279 1 model_repository_manager.cc:594] VersionStates() 'wav2lip'
I0803 16:17:24.141336 1 server.cc:504]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0803 16:17:24.141417 1 server.cc:543]
+-------------+-----------------------------------------------------------------+--------+
| Backend     | Path                                                            | Config |
+-------------+-----------------------------------------------------------------+--------+
| tensorrt    | <built-in>                                                      | {}     |
| pytorch     | /opt/tritonserver/backends/pytorch/libtriton_pytorch.so         | {}     |
| tensorflow  | /opt/tritonserver/backends/tensorflow1/libtriton_tensorflow1.so | {}     |
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {}     |
| openvino    | /opt/tritonserver/backends/openvino/libtriton_openvino.so       | {}     |
+-------------+-----------------------------------------------------------------+--------+

I0803 16:17:24.141433 1 model_repository_manager.cc:570] BackendStates()
I0803 16:17:24.141539 1 server.cc:586]
+---------+---------+-----------------------------------------------------------------------------------------------+
| Model   | Version | Status                                                                                        |
+---------+---------+-----------------------------------------------------------------------------------------------+
| wav2lip | 1       | UNAVAILABLE: Internal: ModuleNotFoundError: No module named 'decorator'                       |
|         |         |                                                                                               |
|         |         | At:                                                                                           |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/_cache.py(40): wrapper         |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/util/utils.py(230): <module>   |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/util/__init__.py(83): <module> |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap>(961): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/notation.py(8): <module>  |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap>(1050): _handle_fromlist                                       |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/convert.py(7): <module>   |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/core/__init__.py(5): <module>  |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap>(1050): _handle_fromlist                                       |
|         |         |   /tmp/python_env_z0IKPf/0/lib/python3.8/site-packages/librosa/__init__.py(211): <module>     |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
|         |         |   /models/wav2lip/1/model.py(13): <module>                                                    |
|         |         |   <frozen importlib._bootstrap>(219): _call_with_frames_removed                               |
|         |         |   <frozen importlib._bootstrap_external>(843): exec_module                                    |
|         |         |   <frozen importlib._bootstrap>(686): _load_unlocked                                          |
|         |         |   <frozen importlib._bootstrap>(975): _find_and_load_unlocked                                 |
|         |         |   <frozen importlib._bootstrap>(991): _find_and_load                                          |
+---------+---------+-----------------------------------------------------------------------------------------------+

I0803 16:17:24.141647 1 tritonserver.cc:1718]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                                                                                  |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                                                                                                 |
| server_version                   | 2.12.0                                                                                                                                                                                 |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0]         | /models                                                                                                                                                                                |
| model_control_mode               | MODE_NONE                                                                                                                                                                              |
| strict_model_config              | 1                                                                                                                                                                                      |
| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                              |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                               |
| min_supported_compute_capability | 6.0                                                                                                                                                                                    |
| strict_readiness                 | 1                                                                                                                                                                                      |
| exit_timeout                     | 30                                                                                                                                                                                     |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0803 16:17:24.141660 1 server.cc:234] Waiting for in-flight requests to complete.
I0803 16:17:24.141666 1 model_repository_manager.cc:694] AsyncUnload() 'wav2lip'
I0803 16:17:24.141671 1 model_repository_manager.cc:988] TriggerNextAction() 'wav2lip' version 1: 2
I0803 16:17:24.141676 1 model_repository_manager.cc:1071] Unload() 'wav2lip' version 1
I0803 16:17:24.141683 1 model_repository_manager.cc:534] LiveBackendStates()
I0803 16:17:24.141692 1 server.cc:249] Timeout 30: Found 0 live models and 0 in-flight non-inference requests
I0803 16:17:24.141699 1 triton_backend_manager.cc:101] unloading backend 'pytorch'
I0803 16:17:24.141707 1 triton_backend_manager.cc:101] unloading backend 'tensorflow'
I0803 16:17:24.141719 1 triton_backend_manager.cc:101] unloading backend 'onnxruntime'
I0803 16:17:24.141761 1 triton_backend_manager.cc:101] unloading backend 'openvino'
error: creating server: Internal - failed to load all models

Issue Analytics

State:
Created 2 years ago
Comments:11 (6 by maintainers)

Top GitHub Comments

2reactions

ghostcommented, Aug 5, 2021

conda does not seem to isolate the environment completely like virtualenv

The local path of a user is included in os.sys.path so conda does not install packages if they are already in there.

A workaround would be exporting the environment variable PYTHONNOUSERSITE before activating conda environment

$ export PYTHONNOUSERSITE=True
$ conda activate test2
$ python
Python 3.8.10 (default, Jun  4 2021, 15:09:15)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from pprint import pprint
>>> pprint(os.sys.path)
['',
 '/home/user/miniconda3/envs/test2/lib/python38.zip',
 '/home/user/miniconda3/envs/test2/lib/python3.8',
 '/home/user/miniconda3/envs/test2/lib/python3.8/lib-dynload',
 '/home/user/miniconda3/envs/test2/lib/python3.8/site-packages']
$ conda deactivate

# Note that the local path of user is included
$ unset PYTHONNOUSERSITE
$ conda activate test2
$ python
Python 3.8.10 (default, Jun  4 2021, 15:09:15)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from pprint import pprint
>>> pprint(os.sys.path)
['',
 '/home/user/miniconda3/envs/test2/lib/python38.zip',
 '/home/user/miniconda3/envs/test2/lib/python3.8',
 '/home/user/miniconda3/envs/test2/lib/python3.8/lib-dynload',
 '/home/user/.local/lib/python3.8/site-packages',
 '/home/user/miniconda3/envs/test2/lib/python3.8/site-packages']

I guess we should mention it in README?

1reaction

Tabriziancommented, Aug 11, 2021

@gioipv Did you set that environment before installing the packages as well?

Top Results From Across the Web

Python error "ImportError: No module named" - Stack Overflow

In my case, I was using sys.path.insert() to import a local module and was getting module not found from a different library. I...

ModuleNotFoundError: no module named Python Error [Fixed]

When you try to import a module in a Python file, Python tries to resolve ... the module not found error as both...

Python Import Error (ModuleNotFoundError) - Finxter

An ImportError is detected when Python has problems with a successful module import. Usually this problem is caused by the incorrect path and...

custom python module not found while using dbx on pycharm

Building a pyspark application using pycharm IDE. I have tested the code in local and wanted to run on databricks cluster from IDE...

module not found error in django project - You.com

I got ModuleNotFoundError: No module named 'graphene_django' error after I added 'graphene_django' in settings.py and run the python manage.py runserver command ...