warp transducer installation error

Describe the issue Getting the following while trying to use install_wrap-rrnt.sh

gcc -pthread -B /home/ubuntu/asr_research/espnet/tools/venv/envs/whale/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ubuntu/asr_research/espnet/tools/venv/envs/whale/include -fPIC -O2 -isystem /home/ubuntu/asr_research/espnet/tools/venv/envs/whale/include -Iusr/local/cuda/include -fPIC -I/home/ubuntu/asr_research/espnet/tools/warp-transducer/include -I/home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include -I/home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include/TH -I/home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include/THC -I/home/ubuntu/asr_research/espnet/tools/venv/envs/whale/include/python3.9 -c src/binding.cpp -o build/temp.linux-x86_64-cpython-39/src/binding.o -fPIC -std=c++14 -DWARPRNNT_ENABLE_GPU -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=warp_rnnt -D_GLIBCXX_USE_CXX11_ABI=0 In file included from /home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include/c10/cuda/CUDAGraphsC10Utils.h:3:0, from /home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include/c10/cuda/CUDACachingAllocator.h:4, from src/binding.cpp:8: /home/ubuntu/asr_research/espnet/tools/venv/envs/whale/lib/python3.9/site-packages/torch/include/c10/cuda/CUDAStream.h:6:10: fatal error: cuda_runtime_api.h: No such file or directory #include <cuda_runtime_api.h> ^~~~~~~~~~~~~~~~~~~~ compilation terminated. error: command ‘/usr/bin/gcc’ failed with exit code 1 [end of output]

Show the check_install.py status by using the following command

cd <espnet-root>/tools
. ./activate_python.sh; python3 check_install.py

[x] python=3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21) [GCC 10.3.0]

Python modules: [x] torch=1.11.0 [x] torch cuda=10.2 [x] torch cudnn=7605 [x] torch nccl [x] chainer=6.0.0 [ ] chainer cuda [ ] chainer cudnn [ ] cupy [x] torchaudio=0.11.0 [x] torch_optimizer=0.3.0 [ ] warpctc_pytorch [ ] warprnnt_pytorch [ ] chainer_ctc [ ] pyopenjtalk [ ] tdmelodic_pyopenjtalk [ ] kenlm [ ] mmseg [x] espnet=202205 [ ] fairseq [ ] phonemizer [ ] gtn [ ] s3prl [x] transformers=4.20.1 [ ] speechbrain [ ] k2 [ ] longformer [ ] nlg-eval [ ] datasets

Executables: [ ] sclite [ ] sph2pipe [ ] PESQ [ ] BeamformIt

Basic environments:

You can obtain them by the following command

cd <espnet-root>/tools
. ./activate_python.sh

echo "- OS information: `uname -mrsv`"
python3 << EOF
import sys, espnet, torch
pyversion = sys.version.replace('\n', ' ')
print(f"""- python version: \`{pyversion}\`
- espnet version: \`espnet {espnet.__version__}\`
- pytorch version: \`pytorch {torch.__version__}\`""")
EOF
cat << EOF
- Git hash: \`$(git rev-parse HEAD)\`
  - Commit date: \`$(git log -1 --format='%cd')\`
EOF

Environments from torch.utils.collect_env: e.g.,

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: CentOS Linux release 7.5.1804 (Core)
GCC version: (GCC) 7.2.0
CMake version: version 2.8.12.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX
GPU 3: TITAN RTX

Nvidia driver version: 440.33.01
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] pytorch-ranger==0.1.1
[pip3] pytorch-wpe==0.0.0
[pip3] torch==1.4.0
[pip3] torch-complex==0.1.1
[pip3] torch-optimizer==0.0.1a14
[pip3] torchaudio==0.4.0
[pip3] warprnnt-pytorch==0.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.1                      217
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.1.0            py37h23d657b_0
[conda] mkl_random                1.1.1            py37h0573a6f_0
[conda] pytorch                   1.4.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] pytorch-wpe               0.0.0                    pypi_0    pypi
[conda] torch-complex             0.1.1                    pypi_0    pypi
[conda] torch-optimizer           0.0.1a14                 pypi_0    pypi
[conda] torchaudio                0.4.0                    pypi_0    pypi
[conda] warprnnt-pytorch          0.1                      pypi_0    pypi

You can obtain them by the following command

cd <espnet-root>/tools
. ./activate_python.sh
python3 -m torch.utils.collect_env

To Reproduce Steps to reproduce the behavior by showing us the specific installation commands with their arguments, e.g.,

cd <espnet-root>/tools
make TH_VERSION=1.3.1

Error logs Paste the error logs. If applicable, add screenshots to help explain your problem.

[b-flo]

Also according to https://github.com/HawkAaron/warp-transducer if you have a non standard CUDA install, add -DCUDA_TOOLKIT_ROOT_DIR=/path/to/cuda option to cmake so that CMake detects CUDA. Not sure why but I need to specify above to overcome basic build issues .

Not sure why but it should be investigated! If that’s the cause, it can be added to the installer. Note that I reinstalled several times warp-transducer in the last past days and didn’t encounter any issue. However, @pengchengguo had to add some exports in the installer to make it work:

export LD_LIBRARY_PATH=$CUDAROOT/lib64:$CUDAROOT/extras/CUPTI/lib64:$LD_LIBRARY_PATH                                                                                    
export LIBRARY_PATH=$CUDAROOT/lib64:$LIBRARY_PATH                                                                                                                       
export CUDA_PATH=$CUDAROOT                                                                                                                                              
export CUDA_HOME=$CUDAROOT                                                                                                                                              
export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME                                                                                                                                 
export CFLAGS="-I$CUDAROOT/include $CFLAGS"                                                                                                                             
export CUDABIN=$CUDAROOT/bin

That’s the ones I shared to @pengchengguo, all of them are in my $PATH.

[karthik19967829]

thanks for the response @kan-bayashi @b-flo . The issue was that I had to another / on AWS instance to fix the path . For some reason it removes the the first slashes so the following works.

export CFLAGS="-I/$CUDAROOT/include $CFLAGS"