ReadTimeoutException on outbound connections

Our set up uses Springboot + Webflux to implement micro services. On receipt of request we transform the request and make outbound requests using WebClient. Our connections are pooled. Below is the typical configuration

Connection Pool creation

private ConnectionProvider connectionProvider() {
        return ConnectionProvider
            .builder("default)
            .maxConnections(500)
            .maxIdleTime(Duration.ofSeconds(30))
            .maxLifeTime(Duration.ofSeconds(60))
            .evictInBackground(Duration.ofSeconds(120))
            .metrics(true)
            .disposeInactivePoolsInBackground(Duration.ofSeconds(180), Duration.ofSeconds(175))
            .build();
    }

HttpClient creation

public HttpClient httpClient() {
        var customLoggingHandler = new HttpLoggingHandler(HttpClient.class);
        var httpClient = HttpClient
            .create(connectionProvider())
            .responseTimeout(Duration.ofMillis(28500))
            .runOn(LoopResources.create("reactor-client-nio")) 
            .doOnRequest((request, connection) -> connection.addHandlerFirst(customLoggingHandler))
            .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 28500);

        SslContext sslContext = sslContextFactory.newSslContext();
        if (sslContext != null) {
            httpClient = httpClient.secure(sslProvider -> sslProvider.sslContext(sslContext));
        }
        return httpClient;
    }

ReactorHttpClientConfiguration

    @Bean
    public ReactorClientHttpConnector standardWebClientConnector(StandardHttpClientConfigProperties config, SslContextFactory sslContextFactory) {
        HttpClient httpClient = new HttpClientFactory(config.getHttpClient(), sslContextFactory).httpClient();
        return new ReactorClientHttpConnector(httpClient);
    }

WebClient creation

    @Bean
    @Qualifier("externalWebClient")
    public WebClient webClient(WebClient.Builder webClientBuilder,
        ReactorClientHttpConnector standardWebClientConnector, ExchangeStrategies exchangeStrategies) {
        return webClientBuilder
            .exchangeStrategies(exchangeStrategies)
            .clientConnector(standardWebClientConnector)
            .build();
    }

In environments where there is less traffic we see ReadTimeoutException if the request takes more than 28500ms but in environments where there is high traffic we see ReadTimeoutException within few seconds of request being made. Below are the logs lines

{"log":"2021-07-03 15:30:21:314 [nt-epoll-3] ERROR .controller.AbstractController a7d8797f-63da-4ab3-aae0-f4d3034afac3 - Unknown exception\n","stream":"stdout","time":"2021-07-03T05:30:21.315151481Z"}
{"log":"org.springframework.web.reactive.function.client.WebClientRequestException: nested exception is io.netty.handler.timeout.ReadTimeoutException\n"

{"log":"2021-07-03 15:30:20:228 [nt-epoll-3] DEBUG efaultPooledConnectionProvider a7d8797f-63da-4ab3-aae0-f4d3034afac3 - [id:77ed51a6-3, L:/10.70.12.98:55550 - R:www.acme.com.au/13.226.106.76:443] onStateChange(GET{uri=/uri/test, connection=PooledConnection{channel=[id: 0x77ed51a6, L:/10.70.12.98:55550 - R:www.acme.com.au/13.226.106.76:443]}}, [request_sent])\n","stream":"stdout","time":"2021-07-03T05:30:20.228483504Z"}

{"log":"2021-07-03 15:30:20:227 [nt-epoll-3] DEBUG efaultPooledConnectionProvider a7d8797f-63da-4ab3-aae0-f4d3034afac3 - [id:77ed51a6-3, L:/10.70.12.98:55550 - R:www.acme.com.au/13.226.106.76:443] onStateChange(GET{uri=/uri/test, connection=PooledConnection{channel=[id: 0x77ed51a6, L:/10.70.12.98:55550 - R:www.acme.com.au/13.226.106.76:443]}}, [request_prepared])\n","stream":"stdout","time":"2021-07-03T05:30:20.228403471Z"}

I have been trying to identify patterns as to when this can happen but no luck so far. We are using reactory-netty version 1.0.7 and springboot version 2.5.0

[violetagg]

@itsnagaraj That’s very interesting. Thanks a lot for the logs.

I think that the issue is that we never observe Removed handler ... but rather Handler [reactor.left.responseTimeoutHandler] already exists in the pipeline ...

I need to check what’s going on.

[itsnagaraj]

After aligning the versions of com.newrelic.telemetry:micrometer-registry-new-relic and com.newrelic.agent.java:newrelic-api with the version of springboot currently used we are no longer seeing the issue in our higher environments.