Improve the performance of methods SoftAssertions#assertThat

Summary

We recently upgraded from AssertJ 3.8.0 to 3.22.0 and we noticed significant slow down in tests using SoftAssertions#assertThat. I look into it and found that AssertJ switched from cglib to ByteBuddy in 3.9.1. I did some simple performance corporation between version and found that proxy generation is ~4x slower compare to AssertJ 3.8.0. with cglib. Speed difference is quite noticeable especially on first running test. TypeCache in SoftProxies is kinda helping, but only tests running after type was already cached. I also compare times from AssertJ 3.22.0 to Mockito 3.2.4 (which is quite old, but it’s too using ByteBuddy to generate sub-classes) and found that execution times for generating classes are almost identical.

This slowdown make SoftAssertions#assertThat unsuitable for some of our tests, as they must run under 2s + it makes harder to predict actual execution time of whole test. We found workaround in SoftAssertions#check(() -> assertThat) but this syntax can be harder to read in longer tests so we would prefer to use SoftAssertions#assertThat as it’s more natural.

I want to ask you if you could please look into this problem and found if there is any space to improvement?

I understand that there isn’t probably any space on ByteBuddy side for improvements, but maybe there could be done something on ours? Maybe change some strategy or use ByteBuddy to generate class from interface (not actual class) and delegate calls to real object (but that would require create interfaces for each Assert class and I’m not sure if it would be possible due generic + if such change wouldn’t result if worse runtime performance). This problem can be also environment problem and sub-class generation is much faster on newer versions of JDK (but I have doubts about that, I experienced similar problems on up-to-date versions of JDK 11 too).

Testing environment

java version “11.0.12” 2021-07-20 LTS Java™ SE Runtime Environment 18.9 (build 11.0.12+8-LTS-237) Java HotSpot™ 64-Bit Server VM 18.9 (build 11.0.12+8-LTS-237, mixed mode)

Tests were performed in Intellij Idea 2020.2.4

AssertJ performance comparison

Used tests:

//used for AssertJ
public class MainTest {
    private final Exception exception = new Exception("lol");
    private final Map<Object, Object> map = Map.of();
    private final Object object = new Object();
    private final List<String> list = List.of();

    @Test
    public void text() { SoftAssertions.assertSoftly(softly -> softly.assertThat("text")); }
    @Test
    public void object() { SoftAssertions.assertSoftly(softly -> softly.assertThat(object)); }
    @Test
    public void list() { SoftAssertions.assertSoftly(softly -> softly.assertThat(list)); }
    @Test
    public void optional() { SoftAssertions.assertSoftly(softly -> softly.assertThat(Optional.empty())); }
    @Test
    public void exception() { SoftAssertions.assertSoftly(softly -> softly.assertThat(exception)); }
    @Test
    public void booleanAssert() { SoftAssertions.assertSoftly(softly -> softly.assertThat(Boolean.FALSE)); }
    @Test
    public void map() { SoftAssertions.assertSoftly(softly -> softly.assertThat(map)); }
}

//Mockito 3.2.4
public class MockitoTest {
    private final Exception exception = new Exception("lol");
    private final Map<Object, Object> map = Map.of();
    private final Object object = new Object();
    private final List<String> list = List.of();

    private static <T> T mock(Class<T> classToMock) { return Mockito.mock(classToMock, InvocationOnMock::callRealMethod); }
    @Test
    public void text() { mock(StringAssert.class); }
    @Test
    public void object() { mock(ObjectAssert.class); }
    @Test
    public void list() { mock(ListAssert.class); }
    @Test
    public void optional() { mock(OptionalAssert.class); }
    @Test
    public void exception() { mock(ThrowableAssert.class); }
    @Test
    public void booleanAssert() { mock(BooleanAssert.class); }
    @Test
    public void map() { mock(MapAssert.class); }
}

Results: AssertJ 3.8.0

AssertJ 3.22.0

Mockito 3.2.4

[raphw]

https://github.com/raphw/byte-buddy/tree/master/byte-buddy-maven-plugin

https://github.com/raphw/byte-buddy/tree/master/byte-buddy-gradle-plugin

[filiphr]

Perhaps an option for AssertJ would be to generate the proxies for the known classes during build time. Not sure what kind of an impact this can have on the JAR size of AssertJ.

Apart from that not sure if @raphw might have some better ideas for how to improve the performance. Perhaps there are some things that can be done in AssertJ, that we don’t know about.