Use ccache

This issue is intended to discuss the use of ccache when building compiled packages and the findings I’ve got on a experimental branch.

Introduction

ccache is a compiler cache. It speeds up recompilation by caching previous compilations and detecting when the same compilation is being done again. Supported languages are C, C++, Objective-C and Objective-C++.

From my personal experience ccache is very reliable and after a couple of years using it at work we never found any issue. Additionally adopting ccache was a huge improvement.

Why

Using ccache on conda-forge could bring a lot of benefits:

• Faster feedback for mantainers of large packages. I personally find it very demotivating trying to make a large package build on CI.
• Possibly decrease CI’s queue size
• Make it possible to build packages that exceed timeout limits

Obtained results

When experimenting with llvmdev-feedstock the results were very promising.

• On CircleCI a build with 99.92% cache hit took 15m as opposed to the usual 1h30m
• On CircleCI a first-time build took about 1h instead of the usual 1h30m. This happens because on CircleCI all package variations are built on the same job.
• On Travis a build with 99.92% cache hit took about 14m instead of the usual 44m (very close to the timeout limit).

Necessary changes

TravisCI

 language: generic
 os: osx
 osx_image: xcode6.4
 
+cache:
+  branch: no-md5deep
+  directories:
+  - $HOME/.ccache
+
+
 env:
   matrix:
     

The branch: no-md5deep is a workaround for https://github.com/travis-ci/travis-ci/issues/7456

CircleCI

   # just to pull each time. #rollondockercaching
   override:
     - docker pull condaforge/linux-anvil
+    - ./ci_support/run_docker_build.sh || touch ./build_failed
+  cache_directories:
+    - "./build_artefacts/.ccache"
 
 test:
   override:
     # Run, test and (if we have a BINSTAR_TOKEN) upload the distributions.
-    - ./ci_support/run_docker_build.sh
+    - if test -f ./build_failed; then exit 1; fi

CircleCI cache is only done for the dependencies phase. So to cache the compilation we need to move the build command. Also, the build command cannot fail during the dependencies phase, otherwise the cache will not be saved. That is why the verification is done later.

ccache setup

This would be added preferably to toolchain activate script:

if [ $(uname) == Linux ]; then
    export CC="ccache gcc"
    export CXX="ccache g++"
    export CCACHE_DIR=/feedstock_root/build_artefacts/.ccache
    # Set max cache size so we don't carry old objects for too long
    ccache -M 400M
else
    export CC="ccache clang"
    export CXX="ccache clang++"
    # Set max cache size so we don't carry old objects for too long
    ccache -M 200M
fi
export CCACHE_BASEDIR="${SRC_DIR}"
ccache -z

Note the ccache -M commands to limit the cache size. This is important because the cache is never automatically deleted by TravisCI and CircleCI. Also we would want to set different sizes for both CI’s as CircleCI builds more than one package per job.

Also ccache needs to be added as dependency, preferably on toolchain package

Recipes

There would be no necessary changes on recipes whose build systems respect CC and CXX variables. However for large packages that could exceed the timeout limit it would be desirable to avoid the timeout killing the build command. On Linux this can be achieved with something like:

timeout 5400 make -j{CPU_COUNT}

Avoiding the timeout limits is important because otherwise the cache is not saved.

Windows

On Windows there is clcache which was inspired by ccache and so is very similar. The problem is that AppVeyor cache mechanism is different. The cache is only enabled for regular branchs, not PR’s. Also it limits the cache size to 1GB per account, which in my opinion make the cache useless for a large organization as conda-forge.

References

• https://circleci.com/docs/1.0/how-cache-works/
• https://docs.travis-ci.com/user/caching/
• https://www.appveyor.com/docs/build-cache/

Pending

• Make it work when someone build locally without run_docker_build.sh script
• Use conda build --no-build-id and avoid setting CCACHE_BASEDIR

[gqmelo]

Using the CCACHE_BASEDIR variable however would shorten __FILE__ to a relative path, breaking the “have enough space for install path” concept of the long build path.

I think this would not be a problem. As you said normally __FILE__ is used for debugging or logging, so it would make much difference if it’s relative. And actually a lot of build systems build the files with a relative path.

Also conda’s relocation feature would not replace __FILE__. On source files this macro would contains a source tree path, which is not available on the final package anyway. On header files the macro won’t be expanded when the header is installed to the prefix. So conda would not detect a hard coded path on either case.

But anyway there may be some weird library that may break with this. So I think it would be better to avoid using CCACHE_BASEDIR altogether and we can do this using conda build --no-build-id.

I agree with you it’s better to test with a few packages first to figure out all this little issues. And then after we are satisfied make ccache default.

I think these would be the steps to implement this:

1 - Create a ccache-toolchain package 2 - Add an option to conda-forge.yml to render the recipe with ccache stuff 3 - Test with packages using a large range of build systems and tools (distutils, cython, cmake, autotools, etc.) 4 - After testing enough, make the changes to toolchain package and make ccache-toolchain a dummy package that just depends on toolchain

[msarahan]

IMHO, it’s worth requesting. It’s pretty compelling to say “hey, look, if you give us more cache space, we’ll use much less compute”