NameError: name 'str2optimizer8bit_blockwise' is not defined

i got this error when running dreambooth training from here

chiron    | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('https'), PosixPath('//github.com/pypa/get-pip/raw/5eaac1050023df1f5c98b173b248c260023f2278/public/get-pip.py')}
chiron    |   warn(
chiron    | CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
chiron    | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
chiron    |   warn(
chiron    | WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
chiron    | CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
chiron    | /usr/local/lib/python3.9/site-packages/bitsandbytes/cextension.py:48: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
chiron    |   warn(
Steps:   0%|          | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]Traceback (most recent call last):
chiron    |   File "/app/train_dreambooth.py", line 592, in <module>
chiron    |     main()
chiron    |   File "/app/train_dreambooth.py", line 560, in main
chiron    |     optimizer.step()
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/optimizer.py", line 140, in step
chiron    |     self.optimizer.step(closure)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
chiron    |     return wrapped(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
chiron    |     return func(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron    |     return func(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 265, in step
chiron    |     self.update_step(group, p, gindex, pindex)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron    |     return func(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 506, in update_step
chiron    |     F.optimizer_update_8bit_blockwise(
chiron    |   File "/usr/local/lib/python3.9/site-packages/bitsandbytes/functional.py", line 858, in optimizer_update_8bit_blockwise
chiron    |     str2optimizer8bit_blockwise[optimizer_name][0](
chiron    | NameError: name 'str2optimizer8bit_blockwise' is not defined
Steps:   0%|          | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]
chiron    | Traceback (most recent call last):
chiron    |   File "/usr/local/bin/accelerate", line 8, in <module>
chiron    |     sys.exit(main())
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
chiron    |     args.func(args)
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 837, in launch_command
chiron    |     simple_launcher(args)
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 354, in simple_launcher
chiron    |     raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

result from python -m bitsandbytes

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = False
COMPUTE_CAPABILITIES_PER_GPU = ['7.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable

name 'str2optimizer32bit' is not defined

run inside docker on VM linux ubuntu 20, 64GB, Intel® Xeon® CPU @ 2.20GHz, Nvidia GV100GL [Tesla V100 SXM2 16GB]

[CharlesFauman]

my build is failing here too, with this error. - WSL2, cuda version V10.1.243

tracing the error, it’s failing on the lib.cadam32bit_g32 line of cextension.py (Attribute error) - lib is not None and torch.cuda.is_available, so it doesn’t reach the generate_instructions call, but my COMPILED_WITH_CUDA gets set to False

the problem is that the evaluate_cuda_setup isn’t working correctly, and my binary gets set to libbitsandbytes_cpu.so, even though I have cuda (including libcudart.do).

Looks like this is because i have cuda at usr/lib/wsl/lib, but my cudart_path gets set to /usr/local/cuda/lib64

setting LD_LIBRARY_PATH=“/usr/lib/wsl/lib/” fixed this for me https://github.com/rapidsai/cudf/issues/10308#issuecomment-1042388289

[ekeric13]

@TimDettmers Does the library need to be named literally “cudart.so”? Can it be named “libcudart.so.11.0”?

I am getting

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!

I have the library installed in /usr/lib/x86_64-linux-gnu/libcudart.so.11.0. I tried setting export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/libcudart.so.11.0 but that didn’t work

edit:

okay I think I got it working if I do export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/