Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

The first getClusterInfo fails after restarting a temporalite cluster in a multi cluster environment

See original GitHub issue

Expected Behavior

I am connecting to two different Temporalite clusters. If I turn a cluster off, getClusterInfo fails. When I turn the cluster back on, getClusterInfo should succeed on the first attempt.

Actual Behavior

When I turn the cluster back on, getClusterInfo fails the first time it is called, and then succeeds after. It does not seem to matter how long this first call is after turning the cluster back on, it still fails. I did not see this same issue on a single cluster setup.

Steps to Reproduce the Problem

Test code + comments:

        WorkflowFacade primary = supplier.getHealthyFacade().orElseThrow();
        // Turn off primary cluster
        WorkflowFacade fallback = supplier.getHealthyFacade().orElseThrow();
        assertNotEquals(primary, fallback);
        // Turn the primary cluster on again
        // another == fallback, where it should be primary, because the health check on primary failed.
        WorkflowFacade another = supplier.getHealthyFacade().orElseThrow();
        // This assertion fails
        assertEquals(primary, another);

Health Check code:

return ListenableFuturesExtra.toCompletableFuture(getService().futureStub()
                    .getClusterInfo(GetClusterInfoRequest.newBuilder().build()))
                    .thenApplyAsync(GetClusterInfoResponse::isInitialized)
                    .orTimeout(2000, TimeUnit.MILLISECONDS);
  1. Set up an environment with two+ Temporal clusters, henceforth referred to as “primary” and “fallback”
  2. Turn on both clusters and getClusterInfo the primary, which will succeed. ✅
  3. Turn off the primary cluster and getClusterInfo the primary, which will fail. ✅
  4. Turn on the primary cluster and getClusterInfo the primary, which will fail. We expect success here. I have waited over 2 minutes with the same result. ❌
  5. getClusterInfo the primary, which will succeed, as expected. ✅

Stack traces: Step 3, after turning off primary:

2022-08-03 17:42:52.862  WARN [,,] 41628 --- [ce="default": 1] io.temporal.internal.worker.Poller       : Failure in poller thread Workflow Poller taskQueue="MY_QUEUE", namespace="default": 1

io.grpc.StatusRuntimeException: UNKNOWN: channel closed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:2803) ~[temporal-serviceclient-1.11.0.jar:na]
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:80) ~[temporal-sdk-1.11.0.jar:na]
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:37) ~[temporal-sdk-1.11.0.jar:na]
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:266) ~[temporal-sdk-1.11.0.jar:na]
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:231) ~[temporal-sdk-1.11.0.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: java.nio.channels.ClosedChannelException: null
	at io.grpc.netty.shaded.io.grpc.netty.Utils.statusFromThrowable(Utils.java:271) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onConnectionError(NettyClientHandler.java:500) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:641) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders0(DefaultHttp2ConnectionEncoder.java:251) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:167) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:53) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:170) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:158) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.createStreamTraced(NettyClientHandler.java:609) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:592) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:326) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel.write(AbstractChannel.java:301) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue$AbstractQueuedCommand.run(WriteQueue.java:213) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue.flush(WriteQueue.java:128) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue.access$000(WriteQueue.java:34) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue$1.run(WriteQueue.java:46) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	... 1 common frames omitted
Caused by: io.grpc.netty.shaded.io.netty.channel.StacklessClosedChannelException: null
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]

Step 3 from calling the health check:

2022-08-03 17:42:52.885 ERROR [,,] 41628 --- [    Test worker] c.me.temp.TemporalWorkflowFacade  : WorkflowFacade reported the cluster as unhealthy:

java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNKNOWN: channel closed
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[na:na]
	at com.me.temp.TemporalWorkflowFacade.isHealthy(TemporalWorkflowFacade.java:249) 
	at com.me.temp.multicluster.MultiClusterSpringBootAppIntegrationTest.testSupplierReturnsFallbackWhenPrimaryIsUnhealthy(MultiClusterSpringBootAppIntegrationTest.java:75) ~[integrationTest/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) ~[spring-test-5.3.18.jar:5.3.18]
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97) ~[spring-test-5.3.18.jar:5.3.18]
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) ~[spring-test-5.3.18.jar:5.3.18]
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190) ~[spring-test-5.3.18.jar:5.3.18]
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) ~[na:na]
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) ~[na:na]
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) ~[na:na]
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) ~[na:na]
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) ~[na:na]
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) ~[na:na]
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) ~[na:na]
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) ~[na:na]
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) ~[na:na]
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182) ~[na:na]
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164) ~[na:na]
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414) ~[na:na]
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) ~[na:na]
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: channel closed
	at io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.temporal.serviceclient.GrpcMetricsInterceptor$MetricsClientCall$1.onClose(GrpcMetricsInterceptor.java:123) ~[temporal-serviceclient-1.11.0.jar:na]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.46.0.jar:1.46.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	... 1 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
	at io.grpc.netty.shaded.io.grpc.netty.Utils.statusFromThrowable(Utils.java:271) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onConnectionError(NettyClientHandler.java:500) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:641) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders0(DefaultHttp2ConnectionEncoder.java:251) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:167) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:53) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:170) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:158) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.createStreamTraced(NettyClientHandler.java:609) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:592) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:326) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel.write(AbstractChannel.java:301) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue$AbstractQueuedCommand.run(WriteQueue.java:213) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue.flush(WriteQueue.java:128) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue.access$000(WriteQueue.java:34) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.grpc.netty.WriteQueue$1.run(WriteQueue.java:46) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	... 1 common frames omitted
Caused by: io.grpc.netty.shaded.io.netty.channel.StacklessClosedChannelException: null
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]

Step 4 after turning primary on:

2022-08-03 17:43:11.243  WARN [,,] 41628 --- [ce="default": 4] io.temporal.internal.worker.Poller       : Failure in poller thread Activity Poller taskQueue="MY_QUEUE", namespace="default": 4

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollActivityTaskQueue(WorkflowServiceGrpc.java:2852) ~[temporal-serviceclient-1.11.0.jar:na]
	at io.temporal.internal.worker.ActivityPollTask.poll(ActivityPollTask.java:99) ~[temporal-sdk-1.11.0.jar:na]
	at io.temporal.internal.worker.ActivityPollTask.poll(ActivityPollTask.java:39) ~[temporal-sdk-1.11.0.jar:na]
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:266) ~[temporal-sdk-1.11.0.jar:na]
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:231) ~[temporal-sdk-1.11.0.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7233
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:na]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[na:na]
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Step 4 calling health check:

2022-08-03 17:43:11.310 ERROR [,,] 41628 --- [    Test worker] c.me.temp.TemporalWorkflowFacade  : WorkflowFacade reported the cluster as unhealthy:

java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[na:na]
	at com.me.temp.TemporalWorkflowFacade.isHealthy(TemporalWorkflowFacade.java:249) 
	at com.me.temp.multicluster.MultiClusterSpringBootAppIntegrationTest.testSupplierReturnsFallbackWhenPrimaryIsUnhealthy(MultiClusterSpringBootAppIntegrationTest.java:78) ~[integrationTest/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) ~[spring-test-5.3.18.jar:5.3.18]
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97) ~[spring-test-5.3.18.jar:5.3.18]
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) ~[spring-test-5.3.18.jar:5.3.18]
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) ~[spring-test-5.3.18.jar:5.3.18]
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.rules.RunRules.evaluate(RunRules.java:20) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) ~[junit-4.13.2.jar:4.13.2]
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413) ~[junit-4.13.2.jar:4.13.2]
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190) ~[spring-test-5.3.18.jar:5.3.18]
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) ~[na:na]
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) ~[na:na]
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) ~[na:na]
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) ~[na:na]
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) ~[na:na]
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) ~[na:na]
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) ~[na:na]
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) ~[na:na]
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) ~[na:na]
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) ~[na:na]
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182) ~[na:na]
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164) ~[na:na]
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414) ~[na:na]
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) ~[na:na]
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542) ~[grpc-stub-1.46.0.jar:1.46.0]
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.46.0.jar:1.46.0]
	at io.temporal.serviceclient.GrpcMetricsInterceptor$MetricsClientCall$1.onClose(GrpcMetricsInterceptor.java:123) ~[temporal-serviceclient-1.11.0.jar:na]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.46.0.jar:1.46.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.46.0.jar:1.46.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	... 1 common frames omitted
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7233
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:na]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[na:na]
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.46.0.jar:1.46.0]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Specifications

Temporalite was installed using go install github.com/temporalio/temporalite/cmd/temporalite@latest

  • Version:
 go/bin » ./temporalite -v
temporal version 1.17.1
  • Platform: Apple M1 Max, macOS 12.4

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
neshieldcommented, Aug 4, 2022

How are you configuring your temporalite instances? Eg. does temporalite start --ephemeral -p 7233 and temporalite start --ephemeral -p 7234 work to reproduce?

That command works to reproduce. I had been doing essentially the same thing: ./temporalite start --ephemeral --namespace default and ./temporalite start --ephemeral --port 7234 --namespace default

4. Turn on the primary cluster and getClusterInfo the primary, which will fail. We expect success here. I have waited over 2 minutes with the same result.

Are you able to reproduce this step via the tctl cluster health command?

If I call tctl cluster health after turning the primary cluster back on, it succeeds:

tctl cluster health                                                       
temporal.api.workflowservice.v1.WorkflowService: SERVING

However, the test still shows the same result. The first getClusterInfo fails with Connection refused: /127.0.0.1:7233.

are you running this program in a new process each time?

Yes, it’s a JUnit test.

0reactions
neshieldcommented, Aug 5, 2022

@cretz

Also, you say you have a “fallback” but all yours steps refer to “primary”,…so “fallback” is not needed in the description to describe your issue? Or is the fallback used in some way?

I thought I could only reproduce this in a two+ cluster environment, even though the “fallback” was never used. However, it seems like the second cluster isn’t actually necessary.WorkflowFacade is a wrapper and a Spring Bean which gets created on startup. So, to narrow things down I managed to reproduce the error without the wrapper and without Spring.

public class SimpleTest {
    private static final WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                    .setTarget("127.0.0.1:7233")
                    .setRpcTimeout(Duration.ofMillis(100000))
                    .build());
    @Test public void testDoNotUseFacade() throws InterruptedException {
        // 1. turn on
        assertTrue(isHealthy(service));
        // 2. turn off
        assertFalse(isHealthy(service));
        int counter = 0;
        boolean isHealthy = false;
        // 3. turn on, doesn't seem to matter how long you wait before starting the loop
        while (!isHealthy) {
            isHealthy = isHealthy(service);
            counter++;
            Thread.sleep(500);
        }
        // This seems to always print 2, even if the sleep is for 500 millis or 2000.
        System.out.println("Cluster returned healthy after numAttempts: " + counter);
    }

    public boolean isHealthy(WorkflowServiceStubs service) {
        try {
            return ListenableFuturesExtra.toCompletableFuture(service.futureStub()
                            .getClusterInfo(GetClusterInfoRequest.newBuilder().build()))
                    .thenApplyAsync(GetClusterInfoResponse::isInitialized)
                    .orTimeout(20, TimeUnit.SECONDS).get();
        } catch (Exception e) {
            System.out.println("Threw exception :" + e.getMessage());
            return false;
        }
    }
}

Here are the full logs for a test run:

16:27:00.705 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
16:27:00.711 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
16:27:00.711 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - Java version: 11
16:27:00.713 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
16:27:00.715 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
16:27:00.716 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
16:27:00.716 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
16:27:00.717 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
16:27:00.717 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @4274d07b
16:27:00.718 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
16:27:00.718 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
16:27:00.739 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - maxDirectMemory: 536870912 bytes (maybe)
16:27:00.739 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /var/folders/ks/0wppmjz10dx67djf4r910hs40000gn/T (java.io.tmpdir)
16:27:00.739 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
16:27:00.740 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - Platform: MacOS
16:27:00.741 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: -1 bytes
16:27:00.741 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
16:27:00.742 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.CleanerJava9 - java.nio.ByteBuffer.cleaner(): available
16:27:00.742 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
16:27:00.807 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 20
16:27:00.819 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
16:27:00.819 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
16:27:00.823 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
16:27:00.823 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
16:27:00.828 [Test worker] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
16:27:01.014 [Test worker] INFO io.temporal.serviceclient.WorkflowServiceStubsImpl - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=127.0.0.1:7233}}
16:27:01.160 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector - -Dio.grpc.netty.shaded.io.netty.leakDetection.level: simple
16:27:01.161 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector - -Dio.grpc.netty.shaded.io.netty.leakDetection.targetRecords: 4
16:27:01.168 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.AbstractByteBuf - -Dio.grpc.netty.shaded.io.netty.buffer.checkAccessible: true
16:27:01.168 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.AbstractByteBuf - -Dio.grpc.netty.shaded.io.netty.buffer.checkBounds: true
16:27:01.169 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector@6d5fe24
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 5
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 5
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
16:27:01.210 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
16:27:01.211 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
16:27:01.212 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.useCacheForAllThreads: true
16:27:01.212 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
16:27:01.268 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.channel.DefaultChannelId - -Dio.netty.processId: 30039 (auto-detected)
16:27:01.270 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: false
16:27:01.270 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
16:27:01.276 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.NetUtilInitializations - Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1%lo0)
16:27:01.277 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.NetUtil - Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. Default: 128
16:27:01.279 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.channel.DefaultChannelId - -Dio.netty.machineId: f4:d4:88:ff:fe:61:1a:a9 (auto-detected)
16:27:01.293 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled
16:27:01.293 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 0
16:27:01.293 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384
16:27:01.314 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
16:27:01.314 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8
16:27:01.314 [grpc-default-executor-0] DEBUG io.grpc.netty.shaded.io.netty.util.Recycler - -Dio.netty.recycler.chunkSize: 32
16:27:01.333 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
16:27:01.341 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
16:27:01.348 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND SETTINGS: ack=false settings={MAX_FRAME_SIZE=16384}
16:27:01.349 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND SETTINGS: ack=true
16:27:01.361 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND SETTINGS: ack=true
16:27:01.374 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: 127.0.0.1:7233, :path: /temporal.api.workflowservice.v1.WorkflowService/GetClusterInfo, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.46.0, client-version: 1.11.0, supported-server-versions: >=0.31.0 <2.0.0, client-name: temporal-java, grpc-accept-encoding: gzip, grpc-timeout: 99710624u] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
16:27:01.382 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=5 bytes=0000000000
16:27:01.385 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=5
16:27:01.386 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND PING: ack=false bytes=145258749040133895
16:27:01.386 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND PING: ack=true bytes=145258749040133895
16:27:01.390 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc] padding=0 endStream=false
16:27:01.398 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND DATA: streamId=3 padding=0 endStream=false length=254 bytes=00000000f90a150a0b74656d706f72616c2d676f12063c322e302e300a150a0b74656d706f72616c2d756912063c332e302e300a190a0f74656d706f72616c2d...
16:27:01.398 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] OUTBOUND PING: ack=false bytes=1234
16:27:01.402 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[grpc-status: 0, grpc-message: ] padding=0 endStream=true
16:27:01.404 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 - R:/127.0.0.1:7233] INBOUND PING: ack=true bytes=1234
16:27:04.731 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 ! R:/127.0.0.1:7233] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:authority: 127.0.0.1:7233, :path: /temporal.api.workflowservice.v1.WorkflowService/GetClusterInfo, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.46.0, client-version: 1.11.0, supported-server-versions: >=0.31.0 <2.0.0, client-name: temporal-java, grpc-accept-encoding: gzip, grpc-timeout: 99999618u] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
16:27:04.735 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 ! R:/127.0.0.1:7233] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=2 length=0 bytes=
16:27:04.739 [grpc-nio-worker-ELG-1-3] DEBUG io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler - [id: 0x5af84be4, L:/127.0.0.1:56236 ! R:/127.0.0.1:7233] Sending GOAWAY failed: lastStreamId '2147483647', errorCode '2', debugData ''. Forcing shutdown of the connection.
io.grpc.netty.shaded.io.netty.channel.StacklessClosedChannelException: null
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
Threw exception :io.grpc.StatusRuntimeException: UNKNOWN: channel closed
Threw exception :io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
16:27:10.112 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
16:27:10.116 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
16:27:10.118 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND SETTINGS: ack=false settings={MAX_FRAME_SIZE=16384}
16:27:10.118 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND SETTINGS: ack=true
16:27:10.119 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND SETTINGS: ack=true
16:27:10.619 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: 127.0.0.1:7233, :path: /temporal.api.workflowservice.v1.WorkflowService/GetClusterInfo, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.46.0, client-version: 1.11.0, supported-server-versions: >=0.31.0 <2.0.0, client-name: temporal-java, grpc-accept-encoding: gzip, grpc-timeout: 99999533u] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
16:27:10.621 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=5 bytes=0000000000
16:27:10.623 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=5
16:27:10.624 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND PING: ack=false bytes=145258749040133895
16:27:10.625 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND PING: ack=true bytes=145258749040133895
16:27:10.626 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc] padding=0 endStream=false
16:27:10.627 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND DATA: streamId=3 padding=0 endStream=false length=254 bytes=00000000f90a150a0b74656d706f72616c2d756912063c332e302e300a190a0f74656d706f72616c2d73657276657212063c322e302e300a160a0c74656d706f...
16:27:10.627 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] OUTBOUND PING: ack=false bytes=1234
16:27:10.629 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[grpc-status: 0, grpc-message: ] padding=0 endStream=true
16:27:10.631 [grpc-nio-worker-ELG-1-7] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x5463f454, L:/127.0.0.1:56261 - R:/127.0.0.1:7233] INBOUND PING: ack=true bytes=1234
16:27:10.895 [grpc-connection-manager-thread-0] DEBUG io.temporal.serviceclient.ChannelManager - Resetting gRPC connection backoff on the gRPC channel ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=127.0.0.1:7233}}
Cluster returned healthy after numAttempts: 2

To clarify @neshield, are you saying that your same client isn’t able to successful call after server starts back up?

That is correct. The first call always fails, and the second one succeeds.

@Spikhalskiy

It looks like the server was up according to your description, but for some reason, the connection attempt was refused. Do we have any proxies in the middle here? Envoy maybe? Or it’s a fully local setup and the SDK connects to Temporal Servers directly?

This is a fully local environment. I’m running Java and temporalite on the same machine. I turned off my VPN just to make sure, and there was no difference in the result.

Given that this doesn’t actually require a multi cluster environment, should the issue be renamed?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Unexpected health check behavior when connecting to ...
The first getClusterInfo fails after restarting a temporalite cluster in a multi cluster environment · Issue #101 · temporalio/temporalite · ...
Read more >
SMC is unable to list applications from one cluster - IBM
Symptom. Symptoms are : - "smcadmin cluster view" command reports the cluster status 'error' as below : Status : error IBM Spectrum Symphony....
Read more >
Multi-cluster Warehouses - Snowflake Documentation
A multi-cluster warehouse is defined by specifying the following properties: ... Auto-resuming a suspended warehouse when new queries are submitted.
Read more >
Error creating workload clusters following the quick start guide
All the AWS provider steps from initialization to generating the workload clusters executed successfully. However the workload cluster is stuck ...
Read more >
Ansible rolling restart multi cluster environment - Stack Overflow
I have a use case where I want to execute a rolling restart in multiple clusters. As the first step, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Jackson configuration is cumbersome

Next page

Workflow.retry that takes a function to decide if retry is needed