Okhttp 3.10.0 Multiple Concurrent POST/PUT Retries

Hello,

We are seeing rare instances of our android app spamming us with requests that look like the following:

h2 2018-09-05T02:38:24.331062Z app/prod-api-alb <userIp> <EndpointIp> 0.003 0.538 0.000 400 400 943 282 "POST https://api... HTTP/2.0" 
h2 2018-09-05T02:38:28.557953Z app/prod-api-alb <userIp> <EndpointIp> 0.000 5.009 0.000 503 503 356 244 "POST https://api... HTTP/2.0" 
h2 2018-09-05T02:38:28.627333Z app/prod-api-alb <userIp> <EndpointIp> 0.000 5.005 0.000 503 503 39 244 "POST https://api... HTTP/2.0"
h2 2018-09-05T02:38:28.629808Z app/prod-api-alb <userIp> <EndpointIp> 0.004 5.003 0.000 503 503 39 244 "POST https://api... HTTP/2.0"

Those 503’s will repeat often hundreds of times essentially DDOS’ing our api. It typically starts with a POST/PUT 400 response to some endpoint, then about 4-5 seconds later 10-20 calls will be made in parallel. These typically all 503 and the cycle continues for quite a while.

I’m assuming this has to do with okhttp retrying failed requests, but I’m unable to duplicate the behavior at all (and it only happens in production one or two times per day. Note that it always seems to occur over http2 as well.

Has anyone seen something like this? Any help is appreciated.

[mtkopone]

Hi. We are also seeing this type of behaviour.

There are bursts of hundreds of POST requests to our backend. It appears that in our backend, the requests time out in the end, while waiting to read the POST body from the request.

We have reviewed our app quite extensively, there should be no reason for this behaviour. As the problem is quite severe, any help would be greatly appreciated.

Environment: react-native app, android 8.0.x, okhttp 3.10.0.

[dmhood]

Just an update, in the 12ish days we’ve had retryOnConnectionFailure set to false, we have yet to see the issue occur again, so it looks like that’s fixed it. It’s not ideal to have it turned off, but its better than spamming our backend at the moment.