Cache `build` directory using GitHub Actions

Describe the bug

I have been trying to implement the build caching through the cache action provided by GitHub Actions. My primary purpose to do this is to restore the build directory and use it to generate only those targets that change in a commit. However, none of my endeavours has been successful so far. I am collaborating with SciPy developers towards migrating the build system to Meson and hence this implementation is necessary to build only those targets that change and drastically reduce the build time over CI.

To Reproduce

The GitHub Actions workflow log can be found here. The GH Action workflow is visible here:

workflow

name: Linux Tests

on:
  push:
    branches:
      - meson
      - master
  pull_request:
    branches:
      - meson
      - master

jobs:
  test_meson:
    name: Meson build
    # If using act to run CI locally the github object does not exist and the usual skipping should not be enforced
    # if: "github.repository != '' || github.repository == 'rgommers/scipy' && !contains(github.event.head_commit.message, '[ci skip]') && !contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip github]') && !contains(github.ref, 'maintenance/') && !contains(github.base_ref, 'maintenance/')"
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.9]

    steps:
    - uses: actions/checkout@v2
      with:
        submodules: recursive

    - name: Setup Python
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install Ubuntu dependencies
      run: |
        # NOTE: not the same OpenBLAS version as in upstream CI (I'm being lazy here)
        sudo apt-get update
        sudo apt-get install -y libopenblas-dev libatlas-base-dev liblapack-dev gfortran libgmp-dev libmpfr-dev libsuitesparse-dev ccache libmpc-dev

    - name: Install Python packages
      run: |
        python -m pip install numpy setuptools wheel cython pytest pytest-xdist pybind11 pytest-xdist mpmath gmpy2 pythran ninja
        python -m pip install git+https://github.com/rgommers/meson.git@fix-cython-mixed-sources

    - name: Install sccache (ubuntu-latest)
      env:
        LINK: https://github.com/mozilla/sccache/releases/download
        SCCACHE_VERSION: v0.2.15
      run: |
        SCCACHE_FILE=sccache-$SCCACHE_VERSION-x86_64-unknown-linux-musl
        mkdir -p $HOME/.local/bin
        curl -L "$LINK/$SCCACHE_VERSION/$SCCACHE_FILE.tar.gz" | tar xz
        mv -f $SCCACHE_FILE/sccache $HOME/.local/bin/sccache
        chmod +x $HOME/.local/bin/sccache
        echo "$HOME/.local/bin" >> $GITHUB_PATH

    - name: Prepare sccache timestamp
      id: sccache_cache_timestamp
      run: |
          NOW=$(date -u +"%F-%T")
          echo "::set-output name=timestamp::${NOW}"
    - name: Cache build
      uses: actions/cache@v2
      env:
        cache-name: cache-build
      with:
        path: .cache/sccache
        key: ${{ runner.os }}-build-${{ env.cache-name }}-sccache-${{ steps.sccache_cache_timestamp.outputs.timestamp }}
        restore-keys: |
          ${{ runner.os }}-build-${{ env.cache-name }}-sccache-

    - name: Cache build dir
      id: cache-scipy-build
      uses: actions/cache@v2
      with:
        path: build
        key: ${{ runner.os }}-${{ matrix.python-version }}

    - name: Build SciPy
      run: |
        ls -l build
        CC='sccache cc' meson setup build --prefix=$PWD/installdir
        CC='sccache cc' ninja -C build -j 2

    - name: Install SciPy
      run: |
        CC='sccache cc' meson install -C build
        sccache --show-stats

    - name: Test SciPy
      run: |
        export PYTHONPATH="$PWD/installdir/lib/python3.9/site-packages/"
        pushd installdir
        python -c "import scipy; import sys; sys.exit(int(not scipy.test(parallel=2)))"
        popd

Expected behavior

The build directory should be configured in such a way that the Meson is able to pick it up and build only the targets that have changed.

I understand that it’s an issue not directly associated with Meson, but GitHub Actions, but I would appreciate any help on this part to improve my understanding of the feasibility of this issue. It will further help me understand if this is possible through Meson itself or not.

system parameters

• Is this a cross build or just a plain native build (for the same computer): plain native build
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.): Ubuntu
• what Python version are you using: 3.9
• what meson --version: 0.59.0
• what ninja --version if it’s a Ninja build: 1.10.0.git.kitware.jobserver-1

[mensinda]

You could try to compare ccache --show-stats between runs to see if the cache is actually being used.

Also, in the linked action it says:

Cache not found for input keys: ccache-linux-release-2021-09-08, ccache-linux-release-2021-09-07

Also, you should really drop the build directory cache since this is just asking for trouble 😃 Meson itself should be fast enough under most circumstances and if it isn’t than this is a bug.

I have never worked with GitHub caching so I am not sure how this could be implemented.

[mensinda]

I don’t think that caching the build directory is a good idea or could possibly work. Meson (or rather ninja) uses timestamps to determine what sources should be rebuilt. So, as soon as the GitHub action checks out the sources all the timestamps will be in the future compared to the cached build directory.

You might have more luck with ccache (which Meson does support natively) and the GitHub ccache action to speed up your build.