Concurrent Bazel test runs silently fail to install ByteBuddy agent

I am transitively using ByteBuddy through mockk in a Bazel project and observed tests to fail when they are run in parallel by Bazel’s test runner due to ByteBuddyAgent.install() silently returning null in all but one of the parallel test runs. On the mockk side, this issue is being tracked as https://github.com/mockk/mockk/issues/627, but I tracked it down to ultimately being caused by this underlying issue.

@raphw This might just as well be a bug in Bazel (or a generally impossible thing to do, cleanly attaching an agent multiple times in parallel), but it is an issue that I would be very interested in hearing your thoughts on. Let me know if I can provide any help from the Bazel side.

A reproducing script which assumes that Bazelisk is installed, which in turn downloads and runs Bazel:

#!/usr/bin/env bash

cat << EOF > .bazelversion
4.2.1
EOF

cat << EOF > WORKSPACE
load("@bazel_tools//tools/build_defs/repo:maven_rules.bzl", "maven_jar")

maven_jar(
    name = "net_bytebuddy_byte_buddy_agent",
    artifact = "net.bytebuddy:byte-buddy-agent:1.11.15",
)

maven_jar(
    name = "junit_junit",
    artifact = "junit:junit:4.12",
)
EOF

cat << EOF > BUILD
java_test(
    name = "test",
    srcs = ["TestByteBuddyAgent.java"],
    test_class = "TestByteBuddyAgent",
    deps = [
        "@net_bytebuddy_byte_buddy_agent//jar",
        "@junit_junit//jar",
    ],
)
EOF

cat << EOF > TestByteBuddyAgent.java
import net.bytebuddy.agent.ByteBuddyAgent;
import org.junit.Test;

import static org.junit.Assert.assertNotEquals;

public class TestByteBuddyAgent {

    @Test
    public void testByteBuddyAgent() {
        assertNotEquals(ByteBuddyAgent.install(), null);
    }
}
EOF

# Passes (single test)
bazelisk test //:test
# Passes (two tests running serially)
bazelisk test //:test --runs_per_test=2 --local_test_jobs=1
# Fails (two tests running in parallel, one fails)
bazelisk test //:test --runs_per_test=2

The logs from the failing test run:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:test
-----------------------------------------------------------------------------
JUnit4 Test Runner
.E
Time: 0.197
There was 1 failure:
1) testByteBuddyAgent(TestByteBuddyAgent)
java.lang.AssertionError: Values should be different. Actual: null
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failEquals(Assert.java:185)
        at org.junit.Assert.assertNotEquals(Assert.java:161)
        at org.junit.Assert.assertNotEquals(Assert.java:175)
        at TestByteBuddyAgent.testByteBuddyAgent(TestByteBuddyAgent.java:10)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:108)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
        at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:116)
        at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:159)
        at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:85)

FAILURES!!!
Tests run: 1,  Failures: 1


BazelTestRunner exiting with a return value of 1
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2021-09-16 13:55:37 --

[raphw]

Yes, Byte Buddy works around the restriction by creating a new process to run the code. If the restriction is lifted by the command line, Byte Buddy more or less runs the same code as in the sample. as it yields the same result, to simplify the test case, I rather used the option in the reproducer.

If you can, please link the ticket you create. I also improved the error handling in Byte Buddy such that null is no longer returned.

[raphw]

Brilliant, thanks for digging!