Build error with --cuda_ext --cpp_ext

I spent the whole day trying to install the new apex extension without success. I was using the old one, then yesterday i wanted to upgrade. But I’m not able to install it.

I’m using Pytorch nighty with conda, with cuda100. I’ve tried with pytoch 1.0.1 and It did not work either.

This is what I get when I run the command from apex directory: python setup.py install --cuda_ext --cpp_ext

Output:

torch.__version__  =  1.1.0.dev20190429

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
from /opt/cuda/bin


Warning:  Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries.

Pytorch binaries were compiled with Cuda 10.0.130

running install
running bdist_egg
running egg_info
writing apex.egg-info/PKG-INFO
writing dependency_links to apex.egg-info/dependency_links.txt
writing top-level names to apex.egg-info/top_level.txt
reading manifest file 'apex.egg-info/SOURCES.txt'
writing manifest file 'apex.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/apex
copying apex/__init__.py -> build/lib.linux-x86_64-3.7/apex
creating build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/__init__.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
creating build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/__init__.py -> build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization
creating build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_initialize.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__version__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp
creating build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/__init__.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN
creating build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/__init__.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
creating build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/__init__.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/optimizers
creating build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/__init__.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/__init__.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
creating build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
running build_ext
building 'apex_C' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/csrc
gcc -pthread -B /home/amar/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include/TH -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include/THC -I/home/amar/miniconda3/include/python3.7m -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /home/amar/miniconda3/compiler_compat -L/home/amar/miniconda3/lib -Wl,-rpath=/home/amar/miniconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

[amaralibey]

Hmm it looks like this is not a Cuda error, but rather, a purely C++ error. It may be unrelated to Apex or even Pytorch for that matter: https://bbs.archlinux.org/viewtopic.php?id=242682. I’m not that familiar with arch linux unfortunately…

You’re right, I had a problem with binutils, i downgraded from 2.32-1 tp 2.31.1-3 luckily Arch linux keeps versions in cache.

I’ve run this command to resolve the problem : sudo pacman -U /var/cache/pacman/pkg/binutils-2.31.1-3-x86_64.pkg.tar.xz

[ck6698000]

mark