BwtB deletes needed runfiles from output base

Description of the problem:

With --remote_download_toplevel, runfiles are not downloaded even if they are missing from the output base under certain circumstances. If a target that is used as data in a bazel test is cached, it will be downloaded for the test, but then deleted right afterward. Then, if you bazel run something that also uses that cached target as data, the bazel server will think the file already exists in the output base, and won’t download it. This leads to broken symlinks in the binary’s runfiles tree.

Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Steps to reproduce: https://gist.github.com/JaredNeil/aaba7e65e22c0a064251f2c2f4e6c6b8 Download zip and run bash -x repro.sh.

What operating system are you running Bazel on?

Ubuntu 18.04

What’s the output of bazel info release?

release 3.7.1

Any other information, logs, or outputs that you want to share?

Full output of running repro.sh from the gist linked above:

$ bash -x repro.sh
+ chmod +x test.sh publish.sh
+ bazelisk build //:data
Starting local Bazel server and connecting to it...
INFO: Invocation ID: dccc83b8-10b1-4651-b4f6-3ad19c2d1902
INFO: Analyzed target //:data (5 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:data up-to-date:
  bazel-bin/data.txt
INFO: Elapsed time: 1.617s, Critical Path: 0.08s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
+ bazelisk clean
INFO: Invocation ID: 185a7ec5-29ed-45fb-a5c3-3bd9d7fe7cad
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
+ bazelisk test //:test
INFO: Invocation ID: 3c3b3346-d8d3-46e2-988c-ab64e12c1e5e
INFO: Analyzed target //:test (22 packages loaded, 229 targets configured).
INFO: Found 1 test target...
Target //:test up-to-date:
  bazel-bin/test
INFO: Elapsed time: 4.144s, Critical Path: 0.11s
INFO: 6 processes: 1 remote cache hit, 3 internal, 2 linux-sandbox.
INFO: Build completed successfully, 6 total actions
//:test                                                                  PASSED in 0.0s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 6 total actions
+ bazelisk run //:publish
INFO: Invocation ID: d87693e1-3cca-4087-beca-63a9290c93db
INFO: Analyzed target //:publish (0 packages loaded, 2 targets configured).
INFO: Found 1 target...
Target //:publish up-to-date:
  bazel-bin/publish
INFO: Elapsed time: 0.160s, Critical Path: 0.01s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions
lrwxrwxrwx 1 jaredneil jaredneil 132 Dec  4 21:43 data.txt -> /home/jaredneil/.cache/bazel/_bazel_jaredneil/e78557b1a5d5d19319bf331b359d3c1d/execroot/__main__/bazel-out/k8-fastbuild/bin/data.txt
Symlink is broken because the target file doesn't exist in the output base
+ echo 'Pubish exit code: 1'
Pubish exit code: 1

[JaredNeil]

Repro also passes on 6.0.0rc1, so I don’t think we need a cherry-pick to make the 6.0.0 release.

[coeuvre]

Thanks for your repro!

The error is not caused by the deletion of the intermediate outputs. Looking at the steps:

1. Build //:data remotely to populate the remote cache.
2. Clean the local workspace and remove data.txt.
3. Test //:test remotely. Bazel won’t download data.txt since the test is executed remotely.
4. Run //:publish but failed since data.txt is not available locally.

The root cause is that Bazel never get a change to download data.txt in this case as downloads from remote cache only happens inside SpawnAction execution and sh_binary is not implemented as SpawnAction.

I am working on separation of downloads which will fix this issue.