cf-java-client 2.11.0 seems to get permanently broken after repeated connection attempts with expired token

This is some kind of regression from earlier behavior. Earlier, I was using client version 2.4.0.RELEASE. And because I was having some unexplained problems. I upgraded to 2.11.0.RELEASE.

However the situation is now a lot worse than before. The problem I was debugging eventually turned out to be not directly related to CF (other than that it was caused by huge-stacktraces dumped to the logs from reactor, which were choking-up the nodejs process that was streaming the logs). After addressing the problem, I found that my 2.11.0 based build was still having a terrible time after experiencing these errors.

In fact, it is so bad that, even after changing the credentials, the client remains unable to connect at all. Yes, even with the correct credentials!

Errors I’m getting at this point are these:

java.lang.IllegalStateException: Timeout on blocking read for 10000 MILLISECONDS
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:106)
	at reactor.core.publisher.Mono.block(Mono.java:1323)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:88)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:91)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
	at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
	at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
	at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:52)
	at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:28)
	at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
	at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
	at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
	at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
	at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
	at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
	at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
	at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
	at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
	at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
	at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
	at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
	at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
	at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
	at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
[cloudfoundry-client-nio-6] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by reactor.ipc.netty.channel.PooledClientContextHandler.operationComplete()
[cloudfoundry-client-nio-5] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by reactor.ipc.netty.channel.PooledClientContextHandler.operationComplete()
reactor.core.Exceptions$BubblingException: java.util.concurrent.TimeoutException: Acquire operation took longer then configured maximum time
	at reactor.core.Exceptions.bubble(Exceptions.java:98)
	at reactor.core.publisher.Operators.onErrorDropped(Operators.java:292)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:152)
	at reactor.ipc.netty.channel.ContextHandler.fireContextError(ContextHandler.java:265)
	at reactor.ipc.netty.channel.PooledClientContextHandler.operationComplete(PooledClientContextHandler.java:149)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer then configured maximum time
	at io.netty.channel.pool.FixedChannelPool.<init>(...)(Unknown Source)
reactor.core.Exceptions$BubblingException: java.util.concurrent.TimeoutException: Acquire operation took longer then configured maximum time
	at reactor.core.Exceptions.bubble(Exceptions.java:98)
	at reactor.core.publisher.Operators.onErrorDropped(Operators.java:292)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:152)
	at reactor.ipc.netty.channel.ContextHandler.fireContextError(ContextHandler.java:265)
	at reactor.ipc.netty.channel.PooledClientContextHandler.operationComplete(PooledClientContextHandler.java:149)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer then configured maximum time
	at io.netty.channel.pool.FixedChannelPool.<init>(...)(Unknown Source)

Any idea what is going on here?

Note: these are the errors near the end… i.e. after we have first done quite a few attempts at connecting with the expired token and finally move on to try a ‘good’ token.

The errors near the beginning are different. Here’s the full log (warning its rather big). https://gist.github.com/kdvolder/fda7915d2b3f198bf6703962250103fb

After downgrading my client back to version 2.4.0.RELEASE everything works perfectly (despite an also enormous error log output 😃

[kdvolder]

@nebhale I retried the scenario that used to break before and it appears to work fine now. I.e. I bombarded the client with requests using expired token firts. Then switched to proper credentials, and it recovered nicely. So as far as I’m concerned, I think you can close this ticket now.

[nebhale]

@kdvolder You should probably do a test run for this just to be sure that we’ve solved your issue, but what we attacked was that the connection pool would block, never releasing connections when retrying. @cwesdorp My test case above would always return an error message as it retired 1,000 times, but the key is that it would retry that 1,000 times unlike before where it would only make it as far as filling the entire connection pool and then block.