Multi-stage build doesn't reuse previously made local cache from Github Cache Action

Troubleshooting

Before sumbitting a bug report please read the Troubleshooting doc.

Behaviour

I am using Multi-stage Dockerfile. There are 3 stages, base, prod and test. I’m also using Github cache action. In the workflow, I build the target base, then prod, then test. This is because I want to run the test first then store local cache. When the tests finished, I want to build base and prod image with push: true to the registry, reusing the previous cache. I tried to control the caching. I put .dockerignore in such a way it ignores everything else except the Dockerfile. This is combined with cache key using build-${{ hashFiles('Dockerfile') }} in order for it to ensure a cache hit.

I’m hoping that with this usage, whenever I change unittests, or github workflow files, the builder will reuse existing cache because obviously the image itself should not change.

However, what happens is:

1. The cache hits (cache key works)
2. Base image build uses the cache (from previous build)
3. Prod image target doesn’t use the cache (even though it’s the same thing, and reuse previous base image cache)
4. Test image target doesn’t use the cache, but this time, I assume because the prod image (the base for test image) was rebuilt.

Even without changing any files and rerun the workflow. This always happens.

Steps to reproduce this issue

1. Create simple Dockerfile with 3 stages, each using previous stage:
2. Make sure .dockerignore filters out all irrelevant files with the build. Ensuring exact cache hit
3. My workflow, using github action cache, and 3 stage build with each stage as separate steps
4. Run the workflow twice at minimum to check if build uses previous cache

Expected behaviour

Second workflow run should build all local images using the cache entirely, because no files are changed. Workflow run after new commit should use cache entirely if build input doesn’t change (files included in the docker context).

Actual behaviour

Base stage uses cache, but prod stage doesn’t use prod cache, however it still uses base stage cache. Test stage uses prod cache, but doesn’t use test stage cache.

Configuration

• Repository URL (if public): https://github.com/lucernae/docker-build-cache-action-test
• Build URL (if public): https://github.com/lucernae/docker-build-cache-action-test/actions/runs/523080913

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Get build cache
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: buildx-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-

      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: test

Logs

logs_17.zip

[lucernae]

Thanks @crazy-max and sorry to necro the thread a little bit.

The GH cache backend works nicely. For others having the same issue, the key solution is to provide a different cache-to location for each different target. If you want to reuse any target, include the possible caches in cache-from (can be multiple lines). The problem is caused by cache invalidation if multiple target uses the same cache-to location.

I’ve made an example action that build all the stages and then reuse it in the same run: https://github.com/lucernae/docker-build-cache-action-test/blob/gh-action-cache-backend/.github/workflows/build-load-test-push.yaml It behaves like this:

• In the same (initial run), different stage caches can be reused (prod), because the test stage also build prod stage
• Second workflow run also reuse the cache, so all the stages uses caches. Very useful if you didn’t change the build. But change other files in repo, like a deployment scripts.

Again, thanks for the quick response.

[crazy-max]

@lucernae Sorry for the delay. There might be some cache invalidation with actions/cache (GC) but I think you’re hitting the following behavior where BuildKit will only build stages that are needed for the final target for your Build all stages and not all stages as you may think (see https://github.com/docker/build-push-action/issues/377#issuecomment-866724261 about the detailed explanations).

So smth like this might be better in your case:

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      
      - name: Get build cache all
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-all-cache
          key: buildx-all-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-all-

      - name: Get build cache base
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-base-cache
          key: buildx-base-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-base-

      - name: Get build cache prod
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-prod-cache
          key: buildx-prod-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-prod-

      - name: Get build cache test
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-test-cache
          key: buildx-test-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-test-

      - name: Build all stages
        id: docker_build_all
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-all-cache-new

      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
            type=local,src=/tmp/.buildx-base-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-base-cache-new
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
            type=local,src=/tmp/.buildx-base-cache
            type=local,src=/tmp/.buildx-prod-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-prod-cache-new
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
            type=local,src=/tmp/.buildx-base-cache
            type=local,src=/tmp/.buildx-prod-cache
            type=local,src=/tmp/.buildx-test-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-prod-cache-new
          target: test
      -
        # Temp fix
        # https://github.com/docker/build-push-action/issues/252
        # https://github.com/moby/buildkit/issues/1896
        name: Move cache
        run: |
          rm -rf /tmp/.buildx-all-cache /tmp/.buildx-base-cache /tmp/.buildx-prod-cache /tmp/.buildx-test-cache
          mv /tmp/.buildx-all-cache-new /tmp/.buildx-all-cache
          mv /tmp/.buildx-base-cache-new /tmp/.buildx-base-cache
          mv /tmp/.buildx-prod-cache-new /tmp/.buildx-prod-cache
          mv /tmp/.buildx-test-cache-new /tmp/.buildx-test-cache

You can also try with the new GitHub Action cache backend:

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      
      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=gha,scope=base
          cache-to: |
            type=gha,scope=base,mode=max
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=gha,scope=prod
            type=gha,scope=base
          cache-to: |
            type=gha,scope=prod,mode=max
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=gha,scope=test
            type=gha,scope=prod
            type=gha,scope=base
          cache-to: |
            type=gha,scope=test,mode=max
          target: test