OperationException - 'Internal Server Error', rpc error: No cluster leader

We are using Spring Cloud (Consul) for service discovery and the consul agent service (3 Node deployment) is deployed with other application service (10 Micro Services) on a AWS Stack.

Recently, we have started noticing intermittent RPC Error, which leads to Re-election of leader and following ERROR is raised from Spring Cloud library, and on monitoring we notice that the service discovery is working as expected and new leader is immeditely elected.

So basically, this ERROR is causing noise in our system.

We were thinking on following lines -

1. Is there a way to only raise this ERROR if the cluster leader is not found by enabling some retry mechanism based on number of tries/ duration ?
2. Any other ERROR which spring consul library would generate if the consul leader is actually not available ? so that we can suppress this ERROR/ raise it as a WARN.

Please suggest.

ERROR ip-XX-XX-XX-XXX | [pool-3-thread-1] org.springframework.cloud.consul.discovery.ConsulCatalogWatch         Error watching Consu
l CatalogServices
"com.ecwid.consul.v1.OperationException: OperationException(statusCode=500, statusMessage='Internal Server Error', statusContent='rpc error: rpc error: No cl
uster leader')
        at com.ecwid.consul.v1.catalog.CatalogConsulClient.getCatalogServices(CatalogConsulClient.java:122)
        at com.ecwid.consul.v1.catalog.CatalogConsulClient.getCatalogServices(CatalogConsulClient.java:108)
        at com.ecwid.consul.v1.ConsulClient.getCatalogServices(ConsulClient.java:320)
        at org.springframework.cloud.consul.discovery.ConsulCatalogWatch.catalogServicesWatch(ConsulCatalogWatch.java:65)
        at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[spencergibb]

Please learn how to properly format code and logs.

consul agent service (3 Node deployment)

This seems odd to me. Do you mean you have a 3 node consul server deployment? Are you pointing spring cloud consul at local agents or at the servers?

[spencergibb]

@gamefundas it’s our recommendation to not point consul clients at the server cluster but at a local agent. There is an open issue to reregister if the agent restarts.