AttributeError: /opt/conda/bin/python: undefined symbol: cudaRuntimeGetVersion

Describe the bug

AttributeError: /opt/conda/bin/python: undefined symbol: cudaRuntimeGetVersion

Reproduction

%cd /workspace

!git clone https://github.com/huggingface/diffusers.git

%cd /workspace/diffusers/examples/dreambooth

!pwd

pip install -U -r requirements.txt

!pip install bitsandbytes

!git config --global credential.helper store

from huggingface_hub import notebook_login
notebook_login()

MODEL_NAME_OR_PATH="CompVis/stable-diffusion-v1-4"
# MODEL_NAME_OR_PATH="runwayml/stable-diffusion-v1-5"
# MODEL_NAME_OR_PATH="runwayml/stable-diffusion-inpainting"
INSTANCE_DIR="./instance_images"
CLASS_DIR="./class_images"
OUTPUT_DIR="./trained_model"
INSTANCE_PROMPT="a photo of sks dog"
CLASS_PROMPT="a photo of dog"
NUM_CLASS_IMAGES=200
MAX_TRAIN_STEPS=800
SEED=1337

# Ensure the directories have been created
!mkdir -p $INSTANCE_DIR
!mkdir -p $CLASS_DIR
!mkdir -p $OUTPUT_DIR

!accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path="$MODEL_NAME_OR_PATH"  \
  --instance_data_dir="$INSTANCE_DIR" \
  --class_data_dir="$CLASS_DIR" \
  --output_dir="$OUTPUT_DIR" \
  --with_prior_preservation --prior_loss_weight=1.0 \
  --instance_prompt="$INSTANCE_PROMPT" \
  --class_prompt="$CLASS_PROMPT" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=2 --gradient_checkpointing \
  --use_8bit_adam \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --num_class_images="$NUM_CLASS_IMAGES" \
  --max_train_steps="$MAX_TRAIN_STEPS" \
  --seed=$SEED

Logs

Traceback (most recent call last):
  File "train_dreambooth.py", line 668, in <module>
    main(args)
  File "train_dreambooth.py", line 454, in main
    import bitsandbytes as bnb
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cextension.py", line 113, in <module>
    lib = CUDASetup.get_instance().lib
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cextension.py", line 109, in get_instance
    cls._instance.initialize()
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cextension.py", line 59, in initialize
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cuda_setup/main.py", line 125, in evaluate_cuda_setup
    cuda_version_string = get_cuda_version(cuda, cudart_path)
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cuda_setup/main.py", line 45, in get_cuda_version
    check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
  File "/opt/conda/lib/python3.7/ctypes/__init__.py", line 377, in __getattr__
    func = self.__getitem__(name)
  File "/opt/conda/lib/python3.7/ctypes/__init__.py", line 382, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))

AttributeError: /opt/conda/bin/python: undefined symbol: cudaRuntimeGetVersion

### System Info

diffusers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• diffusers version: 0.7.2
• Platform: Linux-5.15.0-52-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.13
• PyTorch version (GPU?): 1.13.0+cu117 (True)
• Huggingface_hub version: 0.10.1
• Transformers version: 4.24.0
• Using GPU in script?: RTX 3090
• Using distributed or parallel set-up in script?: No

[0xdevalias]

Just found this, so maybe it’s an issue with bitsandbytes:

• https://github.com/TimDettmers/bitsandbytes/issues/85

Was able to workaround this with setenv LD_LIBRARY_PATH /usr/lib/x86_64-linux-gnu which is where my libcudart.so is (Ubuntu 22.04). Not sure why it misses that.

_Originally posted by @tshadley in https://github.com/TimDettmers/bitsandbytes/issues/85#issuecomment-1304840335_

I had this error and was stumped by it until I realised that my CUDA toolkit had been installed ~incorrectly and was in a place where bitsandbytes couldn’t find it. So when I installed PyTorch and the toolkit via conda instead of pip, the library was able to locate the toolkit and this error went away.

_Originally posted by @65536william in https://github.com/TimDettmers/bitsandbytes/issues/85#issuecomment-1307529648_

[patrickvonplaten]

Hey @0xdevalias,

Yes indeed I think this is a bitsandbytes issue