Task okhttp3.RealCall$AsyncCall@635f40ba rejected from java.util.concurrent.ThreadPoolExecutor@7899bb30[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

Describe the bug

Hi,

I build Kubernetes Client like below

 return new KubernetesClientBuilder().withConfig(new OpenShiftConfigBuilder()
        .withMasterUrl(masterUrl)
        .withOauthToken(authToken)
        .withDisableHostnameVerification(true)
        .withNamespace(null)
        .build()).build();

I experience issues when call the kubernetes apis.

Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@635f40ba rejected from java.util.concurrent.ThreadPoolExecutor@7899bb30[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_202]

I debug the KubernetesClientBuilder.build() method. I think problem is related with clazz.getConstructor(Config.class).newInstance(config); line. It doesn’t configure ThreadPoolExecuter properly.

java.util.concurrent.ThreadPoolExecutor@3cfab340[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

By the way I reviewed API/Impl split title in Migration from 5.x to 6.x doc. But sample usage doesn’t includes withHttpClientFactory(how to create httpclientfactory)

OpenShiftClient openShiftClient = new new KubernetesClientBuilder().withConfig(new OpenShiftConfigBuilder()./*...*/build()).build().adapt(OpenShiftClient.class);

Do you have any suggestion? Is it bug or am i using builder wrongly?

Fabric8 Kubernetes Client version

6.1.1

Steps to reproduce

1. build Kubernetes Client like below

 return new KubernetesClientBuilder().withConfig(new ConfigBuilder()
        .withMasterUrl(masterUrl)
        .withOauthToken(authToken)
        .withDisableHostnameVerification(true)
        .withNamespace(null)
        .build()).build();

1. then call any api

Expected behavior

read configmaps properly

Runtime

OpenShift

Kubernetes API Server version

other (please specify in additional context)

Environment

Windows

Fabric8 Kubernetes Client Logs

at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159) ~[kubernetes-client-api-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:177) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:139) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:88) ~[kubernetes-client-6.1.1.jar:?]
	at com.garanti.ark.batch.scheduler.service.impl.OCPClientServiceImpl.isAppConfigExist(OCPClientServiceImpl.java:126) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.service.impl.BatchServiceImpl.getTaskExecutionCommonArguments(BatchServiceImpl.java:877) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.service.impl.BatchServiceImpl.launchJob(BatchServiceImpl.java:519) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.service.impl.BatchServiceImpl.runService(BatchServiceImpl.java:158) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.rest.contract.impl.JobLauncherControllerImpl.startJob(JobLauncherControllerImpl.java:59) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.rest.contract.impl.JobLauncherControllerImpl$$FastClassBySpringCGLIB$$e5ec9b29.invoke(<generated>) ~[classes/:?]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) ~[spring-context-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.13.jar:5.3.13]
	at com.garanti.ark.batch.scheduler.rest.contract.impl.JobLauncherControllerImpl$$EnhancerBySpringCGLIB$$fff5edd1.startJob(<generated>) ~[classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) [spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) [spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) [spring-webmvc-5.3.13.jar:5.3.13]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) [tomcat-embed-core-9.0.55.jar:4.0.FR]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.13.jar:5.3.13]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) [tomcat-embed-core-9.0.55.jar:4.0.FR]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) [spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.13.jar:5.3.13]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at com.garanti.ark.online.cor.rest.filter.UidFilter.doFilter(UidFilter.java:64) [ark-online-cor-rest-13.1.0-alpha.3.jar:13.1.0-alpha.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50) [spring-cloud-sleuth-core-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at brave.servlet.TracingFilter.doFilter(TracingFilter.java:99) [brave-instrumentation-servlet-5.6.5.jar:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at com.garanti.ark.online.cor.rest.filter.LoggingFilter.doFilter(LoggingFilter.java:65) [ark-online-cor-rest-13.1.0-alpha.3.jar:13.1.0-alpha.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.13.jar:5.3.13]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1722) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.io.InterruptedIOException: executor rejected
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:157) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:144) ~[okhttp-3.14.9.jar:?]
	at okhttp3.RealCall.enqueue(RealCall.java:93) ~[okhttp-3.14.9.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:264) ~[kubernetes-httpclient-okhttp-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:598) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:577) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:469) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:711) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172) ~[kubernetes-client-6.1.1.jar:?]
	... 75 more
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@635f40ba rejected from java.util.concurrent.ThreadPoolExecutor@7899bb30[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_202]
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:154) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:144) ~[okhttp-3.14.9.jar:?]
	at okhttp3.RealCall.enqueue(RealCall.java:93) ~[okhttp-3.14.9.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:264) ~[kubernetes-httpclient-okhttp-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:598) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:577) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:469) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:711) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172) ~[kubernetes-client-6.1.1.jar:?]
	... 75 more

Additional context

Kubernetes version :v1.21.6+b4b4813

[umutcann]

@adietish before client 5.x version, I was calling my createOpenshiftClient method(I shared above) in try block like below.

try (OpenShiftClient openShiftClient = openshiftClientFactory .createOpenshiftClient(ocpClientProperties.getFirstOCP(), ocpClientProperties.getFirstToken())) { this.firstOpenShiftClient = openShiftClient; }

I noticed that connection was closing at the end of try block. I removed try block then worked 😃

[wind57]

It’s likely that you are attempting to use the client after closing it.

This was a great hint!