make ops fails with pytorch compiled from source

Expected results

make ops successfully builds

Actual results

make ops fails to build with the following error

mkdir -p build && cd build && cmake .. && make -j4
CMake Error at CMakeLists.txt:8 (find_package):
  Could not find a package configuration file provided by "Caffe2" with any
  of the following names:

    Caffe2Config.cmake
    caffe2-config.cmake

  Add the installation prefix of "Caffe2" to CMAKE_PREFIX_PATH or set
  "Caffe2_DIR" to a directory containing one of the above files.  If "Caffe2"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/home/ubtuntu/detectron/build/CMakeFiles/CMakeOutput.log".
Makefile:13: recipe for target 'ops' failed
make: *** [ops] Error 1

When exporting the Caffe2 build dir export Caffe2_DIR=/home/ubtuntu/pytorch/build/ The error is

CMake Error at /home/ubtuntu/pytorch/build/Caffe2Config.cmake:14 (include):
  include could not find load file:

    /home/ubtuntu/pytorch/build/public/utils.cmake
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


CMake Error at /home/ubtuntu/pytorch/build/Caffe2Config.cmake:17 (include):
  include could not find load file:

    /home/ubtuntu/pytorch/build/public/threads.cmake
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


CMake Error at /home/ubtuntu/pytorch/build/Caffe2Config.cmake:88 (include):
  include could not find load file:

    /home/ubtuntu/pytorch/build/public/cuda.cmake
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


CMake Error at /home/ubtuntu/pytorch/build/Caffe2Config.cmake:109 (include):
  include could not find load file:

    /home/ubtuntu/pytorch/build/public/mkl.cmake
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


CMake Error at /home/ubtuntu/pytorch/build/Caffe2Config.cmake:112 (include):
  include could not find load file:

    /home/ubtuntu/pytorch/build/Caffe2Targets.cmake
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


CMake Error at /home/ubtuntu/pytorch/build/Caffe2Config.cmake:117 (caffe2_interface_library):
  Unknown CMake command "caffe2_interface_library".
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/ubtuntu/detectron/build/CMakeFiles/CMakeOutput.log".
Makefile:13: recipe for target 'ops' failed
make: *** [ops] Error 1

When exporting the Caffe2 cmake dir export Caffe2_DIR=/home/ubtuntu/pytorch/cmake The error is

CMake Error at /home/ubtuntu/pytorch/cmake/Caffe2Config.cmake:112 (include):
  include could not find load file:

    /home/ubtuntu/pytorch/cmake/Caffe2Targets.cmake
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


CMake Warning (dev) at /home/ubtuntu/pytorch/cmake/public/utils.cmake:20 (get_target_property):
  Policy CMP0045 is not set: Error on non-existent target in
  get_target_property.  Run "cmake --help-policy CMP0045" for policy details.
  Use the cmake_policy command to set the policy and suppress this warning.

  get_target_property() called with non-existent target "caffe2".
Call Stack (most recent call first):
  /home/ubtuntu/pytorch/cmake/Caffe2Config.cmake:117 (caffe2_interface_library)
  CMakeLists.txt:8 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Error at /home/ubtuntu/pytorch/cmake/public/utils.cmake:69 (message):
  You made a CMake build file error: target caffe2 must be of type either
  STATIC_LIBRARY or SHARED_LIBRARY.  However, I got
  __src_target_type-NOTFOUND.
Call Stack (most recent call first):
  /home/ubtuntu/pytorch/cmake/Caffe2Config.cmake:117 (caffe2_interface_library)
  CMakeLists.txt:8 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/ubtuntu/detectron/build/CMakeFiles/CMakeOutput.log".
See also "/home/ubtuntu/detectron/build/CMakeFiles/CMakeError.log".

Detailed steps to reproduce

• ThinkPad T440p GPU: GeForce GT 730M CPU: Intel(R) Core(TM) i5-4300M CPU @ 2.60GHz

• OS

  Distributor ID:	Ubuntu
  Description:	Ubuntu 16.04.5 LTS
  Release:	16.04
  Codename:	xenial
  

1. Update

   sudo apt-get -y update
   

2. Download the .deb files for CUDA 9.0, cuDNN 7.x, and NCCL

3. Install the deb files by running this

   cd ~/Downloads
   
   # add keys
   sudo apt-key add \
   	/var/nccl-repo-2.3.5-ga-cuda9.0/7fa2af80.pub \
   	/var/cuda-repo-9-0-local/7fa2af80.pub \
   
   # install packages
   sudo dpkg -i cuda-repo*.deb *cudnn*.deb nccl*.deb
   sudo apt-get -y update
   sudo apt-get -y install cuda nvidia-cuda-toolkit
   
   # this fices an error with cudnn and gcc5
   # source: https://devtalk.nvidia.com/default/topic/1025801/cudnn-test-did-not-pass/
   sudo sed -i 's:#include "driver_types.h":#include <driver_types.h>:' /usr/include/cudnn.h
   

   REBOOT

   Test CUDA with

   cat /usr/local/cuda/version.txt
   

   Test cuDNN with

   cp -r /usr/src/cudnn_samples_v7/ $HOME/cudnn_test
   cd  $HOME/cudnn_test/mnistCUDNN
   make clean && make
   ./mnistCUDNN
   

4. Install Miniconda

   sudo apt-get -y install wget
   wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
   bash Miniconda3-latest-Linux-x86_64.sh -b -p ~/miniconda
   
   # update .bashrc
   echo 											>> ~/.bashrc
   echo "# Add miniconda to path" 					>> ~/.bashrc 
   echo 'export PATH="$HOME/miniconda/bin:$PATH"' 	>> ~/.bashrc 
   echo 											>> ~/.bashrc
   
   source ~/.bashrc
   

5. Download and build PyTorch

   Dependencies mentioned on PyTorch’s gitpage

   export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" # [anaconda root directory]
   
   # Install basic dependencies
   conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing
   conda install -c mingfeima mkldnn
   
   # Add LAPACK support for the GPU
   conda install -c pytorch magma-cuda90
   

   Dependencies mentioned on Caffe2’s website

   sudo apt-get install -y --no-install-recommends \
         build-essential \
         git \
         libgoogle-glog-dev \
         libgtest-dev \
         libiomp-dev \
         libleveldb-dev \
         liblmdb-dev \
         libopencv-dev \
         libopenmpi-dev \
         libsnappy-dev \
         libprotobuf-dev \
         openmpi-bin \
         openmpi-doc \
         protobuf-compiler \
         
   sudo apt-get install -y --no-install-recommends \
         libgflags-dev \
         cmake
         
   conda install -y future \
         numpy \
         protobuf \
         typing \
         hypothesis
   

   REBOOT

   Clone git repository and build

   PYTORCH=~/pytorch
   git clone --recursive https://github.com/pytorch/pytorch $PYTORCH
   
   cd $PYTORCH
   python setup.py install
   

   Verify

   # To check if Caffe2 build was successful
   python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"
   
   # To check if Caffe2 GPU build was successful
   # This must print a number > 0 in order to use Detectron
   python -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'
   

6. Install COCO

   conda install -y cython
   COCOAPI=~/cocoapi
   git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
   cd $COCOAPI/PythonAPI
   # Install into global site-packages
   make install
   # Alternatively, if you do not have permissions or prefer
   # not to install the COCO API into global site-packages
   python setup.py install --user
   

7. Install Detectron

   REBOOT

   DETECTRON=~/detectron
   git clone https://github.com/facebookresearch/detectron $DETECTRON
   pip install -r $DETECTRON/requirements.txt
   cd $DETECTRON 
   make ops
   

I’ve also tried adding the pytorch/build, pytorch/cmake, and pytorch/caffe2 paths to environment variables Caffe2_DIR and CMAKE_PREFIX_PATH. As well as directly calling cmake passing in the same variables as -D options, and directly modifying the cmake code to point to them.

System information

• Operating system: Ubuntu 16.04 LTS
• Compiler version: gcc/g++ 5.4
• CUDA version: 9.0
• cuDNN version: 7.3
• NVIDIA driver version: NVIDIA-SMI 384.130 Driver Version: 384.130
• PYTHONPATH environment variable: /home/ubtuntu/pytorch/cmake/:/home/ubuntu/miniconda/
• python --version output: Python 3.7.0
• Other Information: the regular make builds successfully.

[linkinpark213]

In my case, adding these two lines at the beginning of Detectron/CMakeLists.txt solved the problem. Please modify the /path/to/pytorch/ part.

set(Caffe2_DIR "/path/to/pytorch/torch/share/cmake/Caffe2/")
include_directories("/path/to/pytorch/torch/lib/include")

[altear]

Others are having this issue https://github.com/pytorch/pytorch/issues/9489#issuecomment-421020215