Official k8s client: Timeout waiting for informers cache to be ready

Using spring-cloud-starter-kubernetes-client-all in my Spring Boot application results in an error message, as well as excessive logging (more details below). Switching to spring-cloud-starter-kubernetes-fabric8-all solves the issues. pom.xml included at the end of this message.

I guess this should not be the case (both clients should not behave too differently). For context, I’m using a Kubernetes cluster hosted by Scaleway, and I’m deploying the application via DevSpace.

The application is running within the default service account, which has all the permissions specified in the documentation. Details at the end of the message.

Related to issue #480 – In case the official client is not currently recommended, shouldn’t this be clearly stated in the documentation?

---

A minute after the application starts, I get the following error:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kubernetesInformerDiscoveryClient' defined in class path resource [org/springframework/cloud/kubernetes/client/discovery/KubernetesDiscoveryClientAutoConfiguration$KubernetesInformerDiscoveryConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.5.jar:5.3.5]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.5.jar:5.3.5]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:769) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1313) ~[spring-boot-2.4.4.jar:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1302) ~[spring-boot-2.4.4.jar:2.4.4]
    at myorg.myapp.Application.main(Application.java:9) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.4.4.jar:2.4.4]
Caused by: java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?
    at org.springframework.cloud.kubernetes.client.discovery.KubernetesInformerDiscoveryClient.afterPropertiesSet(KubernetesInformerDiscoveryClient.java:221) ~[spring-cloud-kubernetes-client-discovery-2.0.2.jar:2.0.2]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1845) ~[spring-beans-5.3.5.jar:5.3.5]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782) ~[spring-beans-5.3.5.jar:5.3.5]
    ... 22 common frames omitted

Additionally, every second the following messages are logged (both before and after the error message above, ad infinitum):

2021-04-18 16:19:12.585  INFO 224 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2021-04-18 16:19:12.935  INFO 224 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2021-04-18 16:19:12.950  INFO 224 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Service#Start listing and watching...

Below I include the required details for reproducibility:

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
      <version>2.4.4</version>
      <relativePath/>
  </parent>

  <groupId>group</groupId>
  <artifactId>artifact</artifactId>
  <version>0.1.0</version>

  <name>Gateway Service</name>

  <properties>
      <java.version>11</java.version>
      <spring-cloud.version>2020.0.2</spring-cloud.version>
      <spring-cloud-kubernetes.version>2.0.2</spring-cloud-kubernetes.version>
      <graphql-java-kickstart.version>11.0.0</graphql-java-kickstart.version>
      <lombok.version>1.18.20</lombok.version>
  </properties>

  <dependencies>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-test</artifactId>
          <scope>test</scope>
      </dependency>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-web</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-actuator</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-security</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.cloud</groupId>
          <artifactId>spring-cloud-starter-kubernetes-client-all</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.cloud</groupId>
          <artifactId>spring-cloud-starter-openfeign</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.kafka</groupId>
          <artifactId>spring-kafka</artifactId>
      </dependency>
      <dependency>
          <groupId>org.springframework.kafka</groupId>
          <artifactId>spring-kafka-test</artifactId>
          <scope>test</scope>
      </dependency>
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-devtools</artifactId>
          <optional>true</optional>
      </dependency>
      <dependency>
          <groupId>org.projectlombok</groupId>
          <artifactId>lombok</artifactId>
          <version>${lombok.version}</version>
          <scope>provided</scope>
      </dependency>
      <dependency>
          <groupId>org.junit.jupiter</groupId>
          <artifactId>junit-jupiter</artifactId>
          <scope>test</scope>
      </dependency>
      <dependency>
          <groupId>com.graphql-java-kickstart</groupId>
          <artifactId>graphql-spring-boot-starter</artifactId>
          <version>${graphql-java-kickstart.version}</version>
      </dependency>
      <dependency>
          <groupId>com.graphql-java-kickstart</groupId>
          <artifactId>graphql-spring-boot-starter-test</artifactId>
          <version>${graphql-java-kickstart.version}</version>
          <scope>test</scope>
      </dependency>
  </dependencies>

  <dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${spring-cloud.version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-kubernetes-dependencies</artifactId>
              <version>${spring-cloud-kubernetes.version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
  </dependencyManagement>

  <build>
      <plugins>
          <plugin>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-maven-plugin</artifactId>
              <configuration>
                  <excludeDevtools>false</excludeDevtools>
              </configuration>
          </plugin>
      </plugins>
  </build>

</project>

Permissions of the service account

$ kubectl describe serviceaccount default
Name:                default
Namespace:           joaomlneto
Labels:              <none>
Annotations:         <none>
Image pull secrets:  devspace-auth-rg-nl-ams-scw-cloud
Mountable secrets:   default-token-2sxvc
Tokens:              default-token-2sxvc
Events:              <none>

$ kubectl describe rolebinding namespace-reader-binding
Name:         namespace-reader-binding
Labels:       <none>
Annotations:  <none>
Role:
Kind:  Role
Name:  namespace-reader
Subjects:
Kind            Name     Namespace
----            ----     ---------
ServiceAccount  default  joaomlneto

$ kubectl describe role namespace-reader
Name:         namespace-reader
Labels:       <none>
Annotations:  <none>
PolicyRule:
Resources              Non-Resource URLs  Resource Names  Verbs
---------              -----------------  --------------  -----
configmaps             []                 []              [list watch get]
endpoints              []                 []              [list watch get]
pods                   []                 []              [list watch get]
secrets                []                 []              [list watch get]
services               []                 []              [list watch get]
configmaps.apps        []                 []              [list watch get]
endpoints.apps         []                 []              [list watch get]
pods.apps              []                 []              [list watch get]
secrets.apps           []                 []              [list watch get]
services.apps          []                 []              [list watch get]
configmaps.extensions  []                 []              [list watch get]
endpoints.extensions   []                 []              [list watch get]
pods.extensions        []                 []              [list watch get]
secrets.extensions     []                 []              [list watch get]
services.extensions    []                 []              [list watch get]

[ryanjbaxter]

This should be fixed in 2020.0.3 https://github.com/spring-cloud/spring-cloud-kubernetes/issues/678

[joaomlneto]

oh,This happened to me, too,How did you solve it in the end ?

Hey @Taogang00 😃

I just used spring-cloud-starter-kubernetes-fabric8-all instead of spring-cloud-starter-kubernetes-client-all and never had problems again. No other changes were required for me.