make ops can't find header files

I solved the previous problem mentioned in https://github.com/facebookresearch/Detectron/issues/634 you can check there for details. But after I successfully sat up Python modules (sudo make), when I was trying to build custom operators library I faced some issues. I solved some but I couldn’t find a solution for this problem. Basically Python can’t find Caffe2 header files.

Error details:

ubuntu@ip-172-31-19-83:~/densepose$ make ops mkdir -p build && cd build && cmake … && make -j4 – Caffe2: Found protobuf with old-style protobuf targets. – Caffe2: Protobuf version – Caffe2: CUDA detected: 9.0 – Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc – Caffe2: CUDA toolkit directory: /usr/local/cuda – Caffe2: Header version is: 9.0 – Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so) – Autodetected CUDA architecture(s): 3.7 – Added CUDA NVCC flags for: -gencode;arch=compute_37,code=sm_37 – Summary: – CMake version : 3.5.1 – CMake command : /usr/bin/cmake – System name : Linux – C++ compiler : /usr/bin/c++ – C++ compiler version : 5.4.0 – CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing – Caffe2 version : 0.8.2 – Caffe2 include path : /home/include – Caffe2 found CUDA : True – CUDA version : 9.0 – CuDNN version : 7.1.4 – Configuring done – Generating done – Build files have been written to: /home/ubuntu/densepose/build make[1]: Entering directory ‘/home/ubuntu/densepose/build’ make[2]: Entering directory ‘/home/ubuntu/densepose/build’ make[3]: Entering directory ‘/home/ubuntu/densepose/build’ make[3]: Entering directory ‘/home/ubuntu/densepose/build’ [ 50%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o [ 50%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o make[3]: Leaving directory ‘/home/ubuntu/densepose/build’ make[3]: Entering directory ‘/home/ubuntu/densepose/build’ [ 50%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o [ 50%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o In file included from /home/ubuntu/densepose/detectron/ops/pool_points_interp.cc:10:0: /home/ubuntu/densepose/detectron/ops/pool_points_interp.h:13:33: fatal error: caffe2/core/context.h: No such file or directory compilation terminated. In file included from /home/ubuntu/densepose/detectron/ops/zero_even_op.cc:9:0: /home/ubuntu/densepose/detectron/ops/zero_even_op.h:12:33: fatal error: caffe2/core/context.h: No such file or directory compilation terminated. CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:86: recipe for target ‘CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o’ failed make[3]: *** [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o] Error 1 make[3]: *** Waiting for unfinished jobs… CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:62: recipe for target ‘CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o’ failed make[3]: *** [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o] Error 1 make[3]: Leaving directory ‘/home/ubuntu/densepose/build’ CMakeFiles/Makefile2:67: recipe for target ‘CMakeFiles/caffe2_detectron_custom_ops.dir/all’ failed make[2]: *** [CMakeFiles/caffe2_detectron_custom_ops.dir/all] Error 2 make[2]: *** Waiting for unfinished jobs… /home/ubuntu/densepose/detectron/ops/zero_even_op.cu:9:37: fatal error: caffe2/core/context_gpu.h: No such file or directory compilation terminated. /home/ubuntu/densepose/detectron/ops/pool_points_interp.cu:11:37: fatal error: caffe2/core/context_gpu.h: No such file or directory compilation terminated. CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:219 (message): Error generating /home/ubuntu/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

CMake Error at caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o.cmake:219 (message): Error generating /home/ubuntu/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:63: recipe for target ‘CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o’ failed make[3]: *** [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o] Error 1 make[3]: *** Waiting for unfinished jobs… CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:70: recipe for target ‘CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o’ failed make[3]: *** [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o] Error 1 make[3]: Leaving directory ‘/home/ubuntu/densepose/build’ CMakeFiles/Makefile2:104: recipe for target ‘CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all’ failed make[2]: *** [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all] Error 2 make[2]: Leaving directory ‘/home/ubuntu/densepose/build’ Makefile:127: recipe for target ‘all’ failed make[1]: *** [all] Error 2 make[1]: Leaving directory ‘/home/ubuntu/densepose/build’ Makefile:13: recipe for target ‘ops’ failed make: *** [ops] Error 2

System information

• Operating system: Ubuntu 16.04
• Compiler version: ?
• CUDA version: CUDA 9
• cuDNN version: 7.1.2
• NVIDIA driver version:
• GPU models (for all devices if they are not all the same): ubuntu@ip-172-31-19-83:~$ nvidia-smi Sat Sep 8 17:14:30 2018
  ±----------------------------------------------------------------------------+ | NVIDIA-SMI 396.37 Driver Version: 396.37 | |-------------------------------±---------------------±---------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 On | 00000000:00:1E.0 Off | 0 | | N/A 36C P8 29W / 149W | 0MiB / 11441MiB | 0% Default | ±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | ±----------------------------------------------------------------------------+

• PYTHONPATH environment variable: /home/ubuntu/pytorch/build
• python --version output: Python 2.7.12

[zaoqiweixaio]

First of all, thank facebook researchers for making their excellent work open. I am new to linux and deep learning, so please forgive me for my unprofessional answer. I finally succeed to run densepose demo. Hope any point helps. I installed caffe2 from source with the following link https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile (tried using anaconda to install at first, but failed for some later operations).

System: Ubuntu 16.04, Cuda 8.0, cuDNN 6.0, git the latest code and related things such as caffe2(pytorch).

Contents added at the end of ~/.bashrc: export PYTHONPATH=/usr/local:$PYTHONPATH export PYTHONPATH=$PYTHONPATH:/home/username/pytorch/build export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/:$LD_LIBRARY_PATH export LIBRARY_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/:$LIBRARY_PATH

export C_INCLUDE_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/include/:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/include/:$CPLUS_INCLUDE_PATH

export C_INCLUDE_PATH=/home/username/pytorch/:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/home/username/pytorch/:$CPLUS_INCLUDE_PATH

export C_INCLUDE_PATH=/home/username/pytorch/build/:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/home/username/pytorch/build/:$CPLUS_INCLUDE_PATH

export C_INCLUDE_PATH=/home/username/pytorch/torch/lib/include/:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/home/username/pytorch/torch/lib/include/:$CPLUS_INCLUDE_PATH

I know it is unprofessional to add so many paths for C and CPlus include path. I get a little crazy when stucking. Whenever cannot finding a .h file, I locate the exact filename (better using the output information, e.g., XX/XX/XX.h) and add a path.

After “make -j4” succeeds, python2 $DENSEPOSE/detectron/tests/test_zero_even_op.py failes. I encounter the error: densepose/build/libcaffe2_detectron_custom_ops_gpu.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl28AllocateAlignedAndAddCleanupEmPFvPvE. Solved by “hyounsamk” in this link https://github.com/facebookresearch/DensePose/issues/119

[vkhalidov]

OK, I see. I suggest that instead of doing sudo make ops, you perform the corresponding commands manually. First, you need to verify that your DensePose sources are clean. Then from your DensePose source directory ~/densepose you need to run the following commands

1. mkdir build
2. cd build
3. cmake .. -DCaffe2_DIR=/usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2
4. make -j4