Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PyTorch 1.9 Support

See original GitHub issue

Hi,

I have the following library / image versions:

  • PyTorch 1.9.0 with CUDA 11.1
  • Based off this image pytorch/pytorch:1.9.0-cuda11.1-cudnn8-devel

When I try pip install warp_rnnt I have the following output:

Error message 
root@a19b333cfe3b:/home/keras/notebook# pip install warp_rnnt
Collecting warp_rnnt
  Using cached warp_rnnt-0.5.0.tar.gz (10 kB)
Requirement already satisfied: pybind11 in /opt/conda/lib/python3.7/site-packages (from warp_rnnt) (2.9.0)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from warp_rnnt) (1.19.5)
Requirement already satisfied: torch>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from warp_rnnt) (1.9.0)
Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.0.0->warp_rnnt) (3.7.4.3)
Building wheels for collected packages: warp-rnnt
  Building wheel for warp-rnnt (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c8
50ea4a028137251/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compil
e(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-73ng5boj
       cwd: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/
  Complete output (106 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/warp_rnnt
  copying warp_rnnt/__init__.py -> build/lib.linux-x86_64-3.7/warp_rnnt
  copying warp_rnnt/test.py -> build/lib.linux-x86_64-3.7/warp_rnnt
  running build_ext
  building 'warp_rnnt._C' extension
  creating /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7
  Emitting ninja build file /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/3] c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fP
IC -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC
-I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.
o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o
  c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/
opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/usr
/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o -DTO
RCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
                   from /usr/include/c++/7/bits/basic_string.h:6361,
                   from /usr/include/c++/7/string:52,
                   from /usr/include/c++/7/stdexcept:39,
                   from /usr/include/c++/7/array:39,
                   from /usr/include/c++/7/tuple:39,
                   from /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp:1:
  /usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
   #include_next <stdlib.h>
                 ^~~~~~~~~~                                                                                                                                                                                                                                    compilation terminated.                                                                                        
  compilation terminated.                                                                                                                                                                                                                                      [2/3] /usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/tem
p.linux-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIN
D11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,co
de=sm_86 -std=c++14
  FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core.o
  /usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-pa
ckages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linu
x-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_CO
MPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_
86 -std=c++14
  In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
                   from /usr/local/cuda/include/crt/common_functions.h:295,
                   from /usr/local/cuda/include/cuda_runtime.h:115,
                   from <command-line>:0:
  /usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
   #include_next <math.h>
                 ^~~~~~~~
  compilation terminated.
  [3/3] /usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/s
ite-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/bu
ild/temp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENS
ION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch
=compute_86,code=sm_86 -std=c++14
  FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core_gather.o
  /usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-pa
ckages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/te
mp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H
'-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compu
te_86,code=sm_86 -std=c++14
  In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
                   from /usr/local/cuda/include/crt/common_functions.h:295,
                   from /usr/local/cuda/include/cuda_runtime.h:115,
                   from <command-line>:0:
  /usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
   #include_next <math.h>
                 ^~~~~~~~
  compilation terminated.
  ninja: build stopped: subcommand failed.  

Traceback (most recent call last):
File “/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1672, in _run_ninja_build
env=env)
File “/opt/conda/lib/python3.7/subprocess.py”, line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.


The above exception was the direct cause of the following exception:


Traceback (most recent call last):
File “<string>”, line 1, in <module>
File “/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py”, line 64, in <module>
“Topic :: Software Development :: Libraries :: Python Modules”,
File “/opt/conda/lib/python3.7/site-packages/setuptools/init.py”, line 153, in setup
return distutils.core.setup(**attrs)
File “/opt/conda/lib/python3.7/distutils/core.py”, line 148, in setup
dist.run_commands()
File “/opt/conda/lib/python3.7/distutils/dist.py”, line 966, in run_commands
self.run_command(cmd)
File “/opt/conda/lib/python3.7/distutils/dist.py”, line 985, in run_command
cmd_obj.run()
File “/opt/conda/lib/python3.7/site-packages/wheel/bdist_wheel.py”, line 290, in run
self.run_command(‘build’)
File “/opt/conda/lib/python3.7/distutils/cmd.py”, line 313, in run_command
self.distribution.run_command(command)
File “/opt/conda/lib/python3.7/distutils/dist.py”, line 985, in run_command
cmd_obj.run()
File “/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py”, line 79, in run
_build_ext.run(self)
File “/opt/conda/lib/python3.7/distutils/command/build_ext.py”, line 340, in run
self.build_extensions()
File “/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 709, in build_extensions
build_ext.build_extensions(self)
File “/opt/conda/lib/python3.7/distutils/command/build_ext.py”, line 449, in build_extensions
self._build_extensions_serial()
File “/opt/conda/lib/python3.7/distutils/command/build_ext.py”, line 474, in _build_extensions_serial
self.build_extension(ext)
File “/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py”, line 196, in build_extension
_build_ext.build_extension(self, ext)
File “/opt/conda/lib/python3.7/distutils/command/build_ext.py”, line 534, in build_extension
depends=ext.depends)
File “/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 539, in unix_wrap_ninja_compile
with_cuda=with_cuda)
File “/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1360, in _write_ninja_file_and_compile_objects
error_prefix=‘Error compiling objects for extension’)
File “/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1682, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension


ERROR: Failed building wheel for warp-rnnt
Running setup.py clean for warp-rnnt
Failed to build warp-rnnt
Installing collected packages: warp-rnnt
Running setup.py install for warp-rnnt … error
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python -u -c ‘import io, os, sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py’“'”‘; file=’“'”‘/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427
c850ea4a028137251/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘, ‘"’"’\n’“'”‘);f.close();exec(comp
ile(code, file, ‘"’“‘exec’”’“‘))’ install --record /tmp/pip-record-vvd1smyc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.7m/warp-rnnt
cwd: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/
Complete output (108 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/warp_rnnt
copying warp_rnnt/init.py -> build/lib.linux-x86_64-3.7/warp_rnnt
copying warp_rnnt/test.py -> build/lib.linux-x86_64-3.7/warp_rnnt
running build_ext
building 'warp_rnnt.C’ extension
creating /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7
Emitting ninja build file /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/build.ninja…
Compiling objects…
Allowing ninja to set a default number of workers… (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,–sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -
fPIC -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/TH
C -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/bindin
g.o -DTORCH_API_INCLUDE_EXTENSION_H ‘-DPYBIND11_COMPILER_TYPE=“gcc"’ '-DPYBIND11_STDLIB="libstdcpp"’ '-DPYBIND11_BUILD_ABI="cxxabi1011"’ -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o
c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,–sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -
I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/u
sr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o -D
TORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"’ '-DPYBIND11_STDLIB="libstdcpp"’ '-DPYBIND11_BUILD_ABI="cxxabi1011"’ -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /usr/include/c++/7/stdexcept:39,
from /usr/include/c++/7/array:39,
from /usr/include/c++/7/tuple:39,
from /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp:1:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next <stdlib.h>
^~~~~~~~~~
compilation terminated.
[2/3] /usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7
/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/
build/temp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options ‘’”’“‘-fPIC’”‘"’’ -DTORCH_API_INCLUDE_EXTE
NSION_H '-DPYBIND11_COMPILER_TYPE="gcc"’ '-DPYBIND11_STDLIB="libstdcpp"’ '-DPYBIND11_BUILD_ABI="cxxabi1011"’ -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=ar
ch=compute_86,code=sm_86 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core_gather.o
/usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-
packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/
temp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS_ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ‘’”‘"’-fPIC’“'”‘’ -DTORCH_API_INCLUDE_EXTENSION_
H ‘-DPYBIND11_COMPILER_TYPE=“gcc"’ '-DPYBIND11_STDLIB="libstdcpp"’ '-DPYBIND11_BUILD_ABI="cxxabi1011"’ -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=com
pute_86,code=sm_86 -std=c++14
In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
from /usr/local/cuda/include/crt/common_functions.h:295,
from /usr/local/cuda/include/cuda_runtime.h:115,
from <command-line>:0:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
#include_next <math.h>
^~~~~~~~
compilation terminated.
[3/3] /usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7
/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/t
emp.linux-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ‘’”’“‘-fPIC’”‘"’’ -DTORCH_API_INCLUDE_EXTENSION_H ‘-DPYB
IND11_COMPILER_TYPE=“gcc"’ '-DPYBIND11_STDLIB="libstdcpp"’ '-DPYBIND11_BUILD_ABI="cxxabi1011"’ -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,
code=sm_86 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core.o
/usr/local/cuda/bin/nvcc  -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-
packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.li
nux-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ‘’”’“‘-fPIC’”‘"’’ -DTORCH_API_INCLUDE_EXTENSION_H ‘-DPYBIND11_
COMPILER_TYPE=“_gcc”’ ‘-DPYBIND11_STDLIB=“_libstdcpp”’ ‘-DPYBIND11_BUILD_ABI=“_cxxabi1011”’ -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=s
m_86 -std=c++14
In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
from /usr/local/cuda/include/crt/common_functions.h:295,
from /usr/local/cuda/include/cuda_runtime.h:115,
from <command-line>:0:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
#include_next <math.h>
^~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File “/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1672, in _run_ninja_build
env=env)
File “/opt/conda/lib/python3.7/subprocess.py”, line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py", line 64, in <module>
    "Topic :: Software Development :: Libraries :: Python Modules",
  File "/opt/conda/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/opt/conda/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/conda/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/opt/conda/lib/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/opt/conda/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/conda/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 709, in build_extensions
    build_ext.build_extensions(self)
  File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
    _build_ext.build_extension(self, ext)
  File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 539, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1360, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1682, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
----------------------------------------



ERROR: Command errored out with exit status 1: /opt/conda/bin/python -u -c ‘import io, os, sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py’“'”‘; file=’“'”‘/tmp/pip-install-b
1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘,
‘"’"’\n’“'”‘);f.close();exec(compile(code, file, ‘"’“‘exec’”’"‘))’ install --record /tmp/pip-record-vvd1smyc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.7m/warp-rnnt Check the logs fo
r full command output.

I used to have similar errors with warp-ctc, and they were resolved with hacks like this:

RUN apt-get install gcc-5 g++-5 g++-5-multilib gfortran-5 -y && \
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5 --slave /usr/bin/gfortran gfortran /usr/bin/gfortran-5 && \
    update-alternatives --query gcc
RUN gcc --version

Is there anything I can do about this without downgrading or changing the current environment?

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
1yticcommented, Jan 5, 2022

I checked this version pip3 install torch==1.9.0+cu111 -f https://download.pytorch.org/whl/cu111/torch_stable.html and warp_rnnt package compiled successfully. I also have the local cuda install with the same version 11.1.

0reactions
snakers4commented, Jan 5, 2022

I see. I tried using these images from scratch:

docker pull pytorch/pytorch:1.10.0-cuda11.3-cudnn8-devel
docker pull pytorch/pytorch:1.9.0-cuda11.1-cudnn8-devel

And it compiled successfully. Looks like the problem is with our images, we most likely borked something up inside of them. Many thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

PyTorch 1.9 Release, including torch.linalg and Mobile ...
Major improvements to support scientific computing, including torch.linalg, torch.special, and Complex Autograd; Major improvements in on-device ...
Read more >
Releases · pytorch/pytorch - GitHub
PyTorch 1.13: beta versions of functorch and improved support for Apple's new M1 ... This release is composed of over 3,400 commits since...
Read more >
How does one install pytorch 1.9 in an HPC that seems to ...
First of all, as @Francois suggested, try to uninstall the CPU only version of pytorch. Also in your installation script, you should use ......
Read more >
PyTorch Release 21.04 - NVIDIA Documentation Center
PyTorch container image version 21.04 is based on 1.9.0a0+2ecb2c7 ... CUDA, PyTorch, and TensorRT are supported in each of the NVIDIA ...
Read more >
torch - PyPI
PyTorch is a Python package that provides two high-level features: ... torch, A Tensor library like NumPy, with strong GPU support.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Strange behavior using PyTorch DDP

Next page

Undefined symbol when loading module