Investigate flaky tests

Hi,

both locally and on CI I encounter relatively frequent flaky tests. While I see most of the time that they’re flaky and ignore them, it happens often enough that I spent/waste time on identifying if those failures are caused by my changes.

Probably not a complete list of things, but things I noticed lately:

ReactiveElasticsearchRepositoriesAutoConfigurationTests > doesNotTriggerDefaultRepositoryDetectionIfCustomized()
ReactiveElasticsearchRepositoriesAutoConfigurationTests > testDefaultRepositoryConfiguration()
DataCassandraTestIntegrationTests > didNotInjectExampleService()
Jetty10ServletWebServerFactoryTests > whenServerIsShuttingDownGracefullyThenResponseToRequestOnIdleConnectionWillHaveAConnectionCloseHeader()
CouchbaseAutoConfigurationIntegrationTests > defaultConfiguration()

Subjectively, the JDK 15 pipeline is a bit flakier, but that might be a false lead.

Anyhow - I wonder if we can do anything about those. I remember that you did an awesome job of increasing timeouts here and there already and tweaked the testcontainer startup attempts, but I think we’re past the testcontainers stage in most of the cases mentioned above.

Cheers, Christoph

[wilkinsona]

There is indeed and it’s really useful. Here are all the test failures over the last 7 days sorted with flaky tests first.

I thought I’d stabilized the Cassandra tests with a timeout increase, but one failed again today with a 10s timeout.

[snicoll]

@daschl also suggested the upgrade to the latest couchbase driver can help. I haven’t seen one flaky test since the upgrade so I am gong to close this one again.