CMake configure run, re-builds conan packages every time

Assumption

I assume that the cmake-conan basically hooks the libraries coming via conan into the cmake build system of my project in a way that once it has been evaluated, i.e. resolving dependencies, download all conan sources for the libs, building them locally, it would not recompute all the steps again if called again.

I compare this to a cmake library target:

add_library(testlib SHARED test.cpp)
...
target_link_libraries(mytool PUBLIC testlib)

In the cmake example above, the library is only built if the input changes. So if you modify test.cpp it is rebuilt. However, if you run ‘make’ again and again without changing test.cpp, it will not rebuild the library.

In contrast, using cmake-conan, it downloads the conan based libraries and builds them every cmake run:

cd build; cmake ..
...
openssl/1.1.1m: Retrieving from server 'conancenter' 
openssl/1.1.1m: Trying with 'conancenter'...
Downloading conanmanifest.txt
Downloading conanfile.py
Downloading conan_export.tgz
openssl/1.1.1m: Downloaded recipe revision 0
conanfile.txt: Installing package

Problem

My cmake-conan integration can install 11 conan packages, builds them locally and then using the cmake find(s) I can use the includes and link to these these.

However, each time I change a tiny line in a CMakeLists.txt it evaluates the conan again and it seems to download the source for each conan packet - again - and build it. Not sure, maybe it also recompiles them. But the overal computation requires like 3 minutes and it seems it also redownloads each packet from the conancenter/jfrog.

How to use the cache and not have to redownload/rebuild each cmake run?

I’ve seen: https://github.com/conan-io/cmake-conan/issues/306 which might be related. I also tried https://github.com/conan-io/cmake-conan#creating-packages but I don’t fully get what this is actually about.

Running cmake, every time:

mkdir build; cmake …; cmake …; cmake …

Will output this each time:

/usr/bin/cmake -DCMAKE_BUILD_TYPE=Debug -G Ninja -S /tmp/someproject-cmake -B /tmp/someproject-cmake/cmake-build-debug-docker-someproject_server
Re-run cmake no build system arguments
-- Conan: Adding jfrog remote repository (https://artifactory.someproject.com/artifactory/api/conan/someproject) verify ssl (True)
-- Conan: Adding conancenter remote repository (https://center.conan.io) verify ssl (True)
-- Conan: checking conan executable
-- Conan: Found program /usr/bin/conan
-- Conan: Version found Conan version 1.47.0
-- Conan executing: /usr/bin/conan install . --remote conancenter --build missing
Auto detecting your dev setup to initialize the default profile (/root/.conan/profiles/default)
CC and CXX: clang, clang++ 
Found clang 10.0
clang>=8, using the major as version
Default settings
	os=Linux
	os_build=Linux
	arch=x86_64
	arch_build=x86_64
	compiler=clang
	compiler.version=10
	compiler.libcxx=libstdc++
	build_type=Release
*** You can change them in /root/.conan/profiles/default ***
*** Or override with -s compiler='other' -s ...s***


Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=clang
compiler.libcxx=libstdc++
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

openssl/1.1.1m: Retrieving from server 'conancenter' 
openssl/1.1.1m: Trying with 'conancenter'...
Downloading conanmanifest.txt
Downloading conanfile.py
Downloading conan_export.tgz
openssl/1.1.1m: Downloaded recipe revision 0
conanfile.txt: Installing package
Requirements
    openssl/1.1.1m from 'conancenter' - Downloaded
Packages
    openssl/1.1.1m:2b1e5ff9df96aaf5924c273e1368c632fcb32dd2 - Download

Installing (downloading, building) binaries...
openssl/1.1.1m: Retrieving package 2b1e5ff9df96aaf5924c273e1368c632fcb32dd2 from remote 'conancenter' 
Downloading conanmanifest.txt
Downloading conaninfo.txt
Downloading conan_package.tgz
openssl/1.1.1m: Package installed 2b1e5ff9df96aaf5924c273e1368c632fcb32dd2
openssl/1.1.1m: Downloaded package revision 0
conanfile.txt: Generator cmake_find_package created FindOpenSSL.cmake
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Aggregating env generators
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo
-- Conan: checking conan executable
-- Conan: Found program /usr/bin/conan
-- Conan: Version found Conan version 1.47.0
-- Conan executing: /usr/bin/conan install . --remote jfrog --build missing --settings build_type=Release --settings compiler=clang --settings compiler.version=10 --settings compiler.libcxx=libstdc++11 --settings compiler.cppstd=17
Configuration:
[settings]


...



Downloading conan_sources.tgz
zlib/1.2.11: Configuring sources in /root/.conan/data/zlib/1.2.11/_/_/source

zlib/1.2.11: Copying sources to build folder
zlib/1.2.11: Building your package in /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2
zlib/1.2.11: Generator cmake created conanbuildinfo.cmake
zlib/1.2.11: Aggregating env generators
zlib/1.2.11: Calling build()
Re-run cmake no build system arguments
-- The C compiler identification is Clang 10.0.1
-- The CXX compiler identification is Clang 10.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: called by CMake conan helper
-- Conan: called inside local cache
-- Conan: Adjusting output directories
-- Conan: Using cmake global configuration
-- Conan: Adjusting language standard
-- Conan: Adjusting fPIC flag (ON)
-- Conan: Compiler Clang>=8, checking major version 10
-- Conan: Checking correct version: 10
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Renaming
--     /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/source_subfolder/zconf.h
-- to 'zconf.h.included' because this file is included with zlib
-- but CMake generates it automatically in the build directory.
-- Configuring done
-- Generating done
-- Build files have been written to: /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder
/usr/bin/cmake -S/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2 -B/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/CMakeFiles /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder//CMakeFiles/progress.marks
/usr/bin/make  -f CMakeFiles/Makefile2 all
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_EXPORT_NO_PACKAGE_REGISTRY
    CMAKE_INSTALL_BINDIR
    CMAKE_INSTALL_DATAROOTDIR
    CMAKE_INSTALL_INCLUDEDIR
    CMAKE_INSTALL_LIBDIR
    CMAKE_INSTALL_LIBEXECDIR
    CMAKE_INSTALL_OLDINCLUDEDIR
    CMAKE_INSTALL_SBINDIR


make[1]: Entering directory '/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder'
/usr/bin/make  -f source_subfolder/CMakeFiles/zlib.dir/build.make source_subfolder/CMakeFiles/zlib.dir/depend
make[2]: Entering directory '/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder'
cd /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2 /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/source_subfolder /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/source_subfolder /root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/source_subfolder/CMakeFiles/zlib.dir/DependInfo.cmake --color=
Dependee "/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/source_subfolder/CMakeFiles/zlib.dir/DependInfo.cmake" is newer than depender "/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/source_subfolder/CMakeFiles/zlib.dir/depend.internal".
Dependee "/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/source_subfolder/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder/source_subfolder/CMakeFiles/zlib.dir/depend.internal".
Scanning dependencies of target zlib
make[2]: Leaving directory '/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder'
/usr/bin/make  -f source_subfolder/CMakeFiles/zlib.dir/build.make source_subfolder/CMakeFiles/zlib.dir/build
make[2]: Entering directory '/root/.conan/data/zlib/1.2.11/_/_/build/2b1e5ff9df96aaf5924c273e1368c632fcb32dd2/build_subfolder'
[ 25%] Building C object source_subfolder/CMakeFiles/zlib.dir/compress.c.o
[ 25%] Building C object source_subfolder/CMakeFiles/zlib.dir/crc32.c.o
[ 31%] Building C object source_subfolder/CMakeFiles/zlib.dir/gzclose.c.o
[ 31%] Building C object source_subfolder/CMakeFiles/zlib.dir/adler32.c.o
[ 31%] Building C object source_subfolder/CMakeFi

CMakeLists.txt

Main CMakeLists.txt

############### conan ################
set(CMAKE_MODULE_PATH ${CMAKE_BINARY_DIR}/conan-cmake/ ${CMAKE_MODULE_PATH})
add_subdirectory(conan-cmake/)

find_package(libpqxx)
find_package(zstd)
find_package(Microsoft.GSL)
find_package(Opus)
find_package(mbedtls)
find_package(nlohmann_json)
find_package(libssh)
find_package(Protobuf)
find_package(ZLIB)
find_package(absl)
############### /conan ################

...

add_subdirectory(conan-cmake/)

list(APPEND CMAKE_MODULE_PATH ${CMAKE_BINARY_DIR})
list(APPEND CMAKE_PREFIX_PATH ${CMAKE_BINARY_DIR})

if(true)
    # standard conan installation, deps will be defined in conanfile.py
    # and not necessary to call conan again, conan is already running
    include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
    conan_basic_setup()
else() # in user space
    # https://github.com/conan-io/cmake-conan
    # https://raw.githubusercontent.com/conan-io/cmake-conan/0.18.1/conan.cmake"
    include(conan.cmake)
    conan_add_remote(NAME jfrog INDEX 0
            URL https://xxx
            VERIFY_SSL True)
    conan_add_remote(NAME conancenter INDEX 1
            URL https://center.conan.io
            VERIFY_SSL True)
    conan_cmake_configure(
            REQUIRES openssl/1.1.1m GENERATORS cmake_find_package # Override to newer version to resolve mismatch between protobuf and libssh
    )
    conan_cmake_install(PATH_OR_REFERENCE . BUILD missing REMOTE conancenter SETTINGS ${settings})
    conan_cmake_configure(
            REQUIRES libpqxx/7.7.0 GENERATORS cmake_find_package
            REQUIRES zstd/1.5.2 GENERATORS cmake_find_package
            REQUIRES ms-gsl/4.0.0 GENERATORS cmake_find_package cmake_find_package_multi
            REQUIRES opus/1.3.1 GENERATORS cmake_find_package
            REQUIRES nlohmann_json/3.10.5 GENERATORS cmake_find_package
            REQUIRES protobuf/3.21.1 GENERATORS cmake_find_package
            REQUIRES abseil/20211102.0 GENERATORS cmake_find_package
            REQUIRES zlib/1.2.11 GENERATORS cmake_find_package # Override to newer version to resolve mismatch between protobuf and libssh
    )

    conan_cmake_autodetect(settings)
    conan_cmake_install(PATH_OR_REFERENCE . BUILD missing REMOTE jfrog SETTINGS ${settings})
endif()

[prince-chrismc]

Unfortunately you need to create the docker container or image with the notion of volumes.

It’s not something I know you can do retroactively

[qknight]

@prince-chrismc added it, wanted to post it right away but managed to forget it (see above, I updated the issue).