Socket closed when trying to use OpenshiftApi

Hi, we are currently trying to set up an operator. While using the “Webserver” example we tried creating a route as well, which is currently not working for us. Trying to create any resource using the KubernetesClient is working, but as soon as we want to create something that is using the OpenShift APi (e.g. listing/creating projects, imagestreams, routes) causes the following exception:

{"@timestamp":"2020-02-03T10:19:43.563+00:00","@version":1,"message":"Exec Failure","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp WebSocket https://172.30.0.1/...","level":"WARN","level_value":30000,"stack_trace":"j.n.SocketException: Socket closed\n\tat j.n.SocketOutputStream.socketWrite(SocketOutputStream.java:118)\n\tat j.n.SocketOutputStream.write(SocketOutputStream.java:155)\n\tat s.s.s.OutputRecord.writeBuffer(OutputRecord.java:431)\n\tat s.s.s.OutputRecord.write(OutputRecord.java:417)\n\tat s.s.s.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)\n\tat s.s.s.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)\n\tat s.s.s.AppOutputStream.write(AppOutputStream.java:123)\n\tat okio.Okio$1.write(Okio.java:79)\n\tat o.AsyncTimeout$1.write(AsyncTimeout.java:180)\n\tat o.RealBufferedSink.flush(RealBufferedSink.java:224)\n\tat o.i.w.WebSocketWriter.writeControlFrame(WebSocketWriter.java:146)\n\tat o.i.w.WebSocketWriter.writePing(WebSocketWriter.java:76)\n\tat o.i.w.RealWebSocket.writePingFrame(RealWebSocket.java:552)\n\tat o.i.w.RealWebSocket$PingRunnable.run(RealWebSocket.java:529)\n\tat j.u.c.Executors$RunnableAdapter.call(Executors.java:511)\n\tat j.u.c.FutureTask.runAndReset(FutureTask.java:308)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\t... 3 frames excluded\n","build_git_commit":"d057398d4d16800b2bdbb3fd6ea16ebbaa8b8e6f","build_version":"1.0.97504","build_timestamp":"2020-02-03T10:09:58Z","build_artifact_id":"k5-operator-application","caller_class_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1","caller_method_name":"onFailure","caller_file_name":"WatchConnectionManager.java","caller_line_number":206,"stack_hash":"2c169fe9"}
{"@timestamp":"2020-02-03T10:19:44.567+00:00","@version":1,"message":"Exec Failure","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor for Watch 1203069976","level":"WARN","level_value":30000,"stack_trace":"j.u.c.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@3e3e76b6 rejected from java.util.concurrent.ThreadPoolExecutor@30f98f01[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]\n\tat j.u.c.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)\n\tat j.u.c.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)\n\tat j.u.c.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:154)\n\t... 16 common frames omitted\nWrapped by: j.i.InterruptedIOException: executor rejected\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:157)\n\tat o.Dispatcher.promoteAndExecute(Dispatcher.java:204)\n\tat o.Dispatcher.enqueue(Dispatcher.java:144)\n\tat o.RealCall.enqueue(RealCall.java:93)\n\tat o.i.w.RealWebSocket.connect(RealWebSocket.java:190)\n\tat o.OkHttpClient.newWebSocket(OkHttpClient.java:409)\n\tat i.f.k.c.d.i.WatchConnectionManager.runWatch(WatchConnectionManager.java:158)\n\tat i.f.k.c.d.i.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)\n\tat i.f.k.c.d.i.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:321)\n\tat i.f.k.c.d.i.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:410)\n\tat j.u.c.Executors$RunnableAdapter.call(Executors.java:511)\n\tat j.u.c.FutureTask.run(FutureTask.java:266)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\t... 3 frames excluded\n","build_git_commit":"d057398d4d16800b2bdbb3fd6ea16ebbaa8b8e6f","build_version":"1.0.97504","build_timestamp":"2020-02-03T10:09:58Z","build_artifact_id":"k5-operator-application","caller_class_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1","caller_method_name":"onFailure","caller_file_name":"WatchConnectionManager.java","caller_line_number":206,"stack_hash":"1332d0ba"}
{"@timestamp":"2020-02-03T10:19:46.568+00:00","@version":1,"message":"Exec Failure","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor for Watch 1203069976","level":"WARN","level_value":30000,"stack_trace":"j.u.c.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@34c61dc6 rejected from java.util.concurrent.ThreadPoolExecutor@30f98f01[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]\n\tat j.u.c.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)\n\tat j.u.c.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)\n\tat j.u.c.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:154)\n\t... 16 common frames omitted\nWrapped by: j.i.InterruptedIOException: executor rejected\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:157)\n\tat o.Dispatcher.promoteAndExecute(Dispatcher.java:204)\n\tat o.Dispatcher.enqueue(Dispatcher.java:144)\n\tat o.RealCall.enqueue(RealCall.java:93)\n\tat o.i.w.RealWebSocket.connect(RealWebSocket.java:190)\n\tat o.OkHttpClient.newWebSocket(OkHttpClient.java:409)\n\tat i.f.k.c.d.i.WatchConnectionManager.runWatch(WatchConnectionManager.java:158)\n\tat i.f.k.c.d.i.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)\n\tat i.f.k.c.d.i.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:321)\n\tat i.f.k.c.d.i.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:410)\n\tat j.u.c.Executors$RunnableAdapter.call(Executors.java:511)\n\tat j.u.c.FutureTask.run(FutureTask.java:266)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\t... 3 frames excluded\n","build_git_commit":"d057398d4d16800b2bdbb3fd6ea16ebbaa8b8e6f","build_version":"1.0.97504","build_timestamp":"2020-02-03T10:09:58Z","build_artifact_id":"k5-operator-application","caller_class_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1","caller_method_name":"onFailure","caller_file_name":"WatchConnectionManager.java","caller_line_number":206,"stack_hash":"1332d0ba"}
{"@timestamp":"2020-02-03T10:19:47.934+00:00","@version":1,"message":"Error on resource: pipeline-manager-from-crd","logger_name":"com.github.containersolutions.operator.EventDispatcher","thread_name":"OkHttp https://172.30.0.1/...","level":"ERROR","level_value":40000,"stack_trace":"j.i.InterruptedIOException: interrupted\n\tat okio.Timeout.throwIfReached(Timeout.java:146)\n\tat okio.Okio$1.write(Okio.java:76)\n\tat o.AsyncTimeout$1.write(AsyncTimeout.java:180)\n\tat o.RealBufferedSink.flush(RealBufferedSink.java:224)\n\tat o.i.h.Http1ExchangeCodec.finishRequest(Http1ExchangeCodec.java:190)\n\tat o.i.c.Exchange.finishRequest(Exchange.java:101)\n\tat o.i.h.CallServerInterceptor.intercept(CallServerInterceptor.java:86)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.c.ConnectInterceptor.intercept(ConnectInterceptor.java:43)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:117)\n\tat o.i.c.CacheInterceptor.intercept(CacheInterceptor.java:94)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:117)\n\tat o.i.h.BridgeInterceptor.intercept(BridgeInterceptor.java:93)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.h.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:117)\n\tat i.f.k.c.u.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:117)\n\tat i.f.k.c.u.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:142)\n\tat o.i.h.RealInterceptorChain.proceed(RealInterceptorChain.java:117)\n\t... 6 frames truncated\n\t... 25 common frames omitted\nWrapped by: i.f.k.c.KubernetesClientException: An error has occurred.\n\tat i.f.k.c.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)\n\tat i.f.k.c.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)\n\tat i.f.o.c.OpenshiftAdapterSupport.isOpenShiftAPIGroups(OpenshiftAdapterSupport.java:80)\n\tat i.f.o.c.OpenShiftConfig.isOpenShiftAPIGroups(OpenShiftConfig.java:121)\n\tat i.f.o.c.d.i.OpenShiftOperation.wrap(OpenShiftOperation.java:48)\n\tat i.f.o.c.d.i.OpenShiftOperation.<init>(OpenShiftOperation.java:40)\n\tat i.f.o.c.d.i.RouteOperationsImpl.<init>(RouteOperationsImpl.java:48)\n\tat i.f.o.c.d.i.RouteOperationsImpl.newInstance(RouteOperationsImpl.java:57)\n\tat i.f.o.c.d.i.RouteOperationsImpl.newInstance(RouteOperationsImpl.java:41)\n\tat i.f.k.c.d.b.BaseOperation.withName(BaseOperation.java:258)\n\tat i.f.k.c.d.b.BaseOperation.createOrReplace(BaseOperation.java:382)\n\tat d.k.c.o.k.r.d.p.PipelineManagerController.createOrUpdateResource(PipelineManagerController.java:100)\n\tat d.k.c.o.k.r.d.p.PipelineManagerController.createOrUpdateResource(PipelineManagerController.java:24)\n\tat c.g.c.o.EventDispatcher.handleEvent(EventDispatcher.java:61)\n\tat c.g.c.o.EventDispatcher.eventReceived(EventDispatcher.java:44)\n\tat c.g.c.o.EventDispatcher.eventReceived(EventDispatcher.java:14)\n\tat i.f.k.c.u.WatcherToggle.eventReceived(WatcherToggle.java:49)\n\tat i.f.k.c.d.i.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:237)\n\tat o.i.w.RealWebSocket.onReadMessage(RealWebSocket.java:322)\n\tat o.i.w.WebSocketReader.readMessageFrame(WebSocketReader.java:219)\n\tat o.i.w.WebSocketReader.processNextFrame(WebSocketReader.java:105)\n\tat o.i.w.RealWebSocket.loopReader(RealWebSocket.java:273)\n\tat o.i.w.RealWebSocket$1.onResponse(RealWebSocket.java:209)\n\tat o.RealCall$AsyncCall.execute(RealCall.java:174)\n\tat o.i.NamedRunnable.run(NamedRunnable.java:32)\n\t... 3 frames truncated\n","build_git_commit":"d057398d4d16800b2bdbb3fd6ea16ebbaa8b8e6f","build_version":"1.0.97504","build_timestamp":"2020-02-03T10:09:58Z","build_artifact_id":"k5-operator-application","caller_class_name":"com.github.containersolutions.operator.EventDispatcher","caller_method_name":"eventReceived","caller_file_name":"EventDispatcher.java","caller_line_number":47,"stack_hash":"dbb396d5"}
{"@timestamp":"2020-02-03T10:19:50.569+00:00","@version":1,"message":"Exec Failure","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor for Watch 1203069976","level":"WARN","level_value":30000,"stack_trace":"j.u.c.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@71c4853a rejected from java.util.concurrent.ThreadPoolExecutor@30f98f01[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]\n\tat j.u.c.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)\n\tat j.u.c.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)\n\tat j.u.c.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:154)\n\t... 16 common frames omitted\nWrapped by: j.i.InterruptedIOException: executor rejected\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:157)\n\tat o.Dispatcher.promoteAndExecute(Dispatcher.java:204)\n\tat o.Dispatcher.enqueue(Dispatcher.java:144)\n\tat o.RealCall.enqueue(RealCall.java:93)\n\tat o.i.w.RealWebSocket.connect(RealWebSocket.java:190)\n\tat o.OkHttpClient.newWebSocket(OkHttpClient.java:409)\n\tat i.f.k.c.d.i.WatchConnectionManager.runWatch(WatchConnectionManager.java:158)\n\tat i.f.k.c.d.i.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)\n\tat i.f.k.c.d.i.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:321)\n\tat i.f.k.c.d.i.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:410)\n\tat j.u.c.Executors$RunnableAdapter.call(Executors.java:511)\n\tat j.u.c.FutureTask.run(FutureTask.java:266)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\t... 3 frames excluded\n","build_git_commit":"d057398d4d16800b2bdbb3fd6ea16ebbaa8b8e6f","build_version":"1.0.97504","build_timestamp":"2020-02-03T10:09:58Z","build_artifact_id":"k5-operator-application","caller_class_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1","caller_method_name":"onFailure","caller_file_name":"WatchConnectionManager.java","caller_line_number":206,"stack_hash":"1332d0ba"}
{"@timestamp":"2020-02-03T10:19:58.570+00:00","@version":1,"message":"Exec Failure","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor for Watch 1203069976","level":"WARN","level_value":30000,"stack_trace":"j.u.c.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@3c82d672 rejected from java.util.concurrent.ThreadPoolExecutor@30f98f01[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]\n\tat j.u.c.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)\n\tat j.u.c.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)\n\tat j.u.c.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:154)\n\t... 16 common frames omitted\nWrapped by: j.i.InterruptedIOException: executor rejected\n\tat o.RealCall$AsyncCall.executeOn(RealCall.java:157)\n\tat o.Dispatcher.promoteAndExecute(Dispatcher.java:204)\n\tat o.Dispatcher.enqueue(Dispatcher.java:144)\n\tat o.RealCall.enqueue(RealCall.java:93)\n\tat o.i.w.RealWebSocket.connect(RealWebSocket.java:190)\n\tat o.OkHttpClient.newWebSocket(OkHttpClient.java:409)\n\tat i.f.k.c.d.i.WatchConnectionManager.runWatch(WatchConnectionManager.java:158)\n\tat i.f.k.c.d.i.WatchConnectionManager.access$1200(WatchConnectionManager.java:48)\n\tat i.f.k.c.d.i.WatchConnectionManager$2$1.execute(WatchConnectionManager.java:321)\n\tat i.f.k.c.d.i.WatchConnectionManager$NamedRunnable.run(WatchConnectionManager.java:410)\n\tat j.u.c.Executors$RunnableAdapter.call(Executors.java:511)\n\tat j.u.c.FutureTask.run(FutureTask.java:266)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\t... 3 frames excluded\n","build_git_commit":"d057398d4d16800b2bdbb3fd6ea16ebbaa8b8e6f","build_version":"1.0.97504","build_timestamp":"2020-02-03T10:09:58Z","build_artifact_id":"k5-operator-application","caller_class_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1","caller_method_name":"onFailure","caller_file_name":"WatchConnectionManager.java","caller_line_number":206,"stack_hash":"1332d0ba"}

PipelineManagerController.java

@Controller(customResourceClass = PipelineManager.class, kind = PipelineManagerController.KIND, crdName = "pipelinemanagers.sample.javaoperatorsdk", group = PipelineManagerController.GROUP, customResourceListClass = PipelineManagerList.class, customResourceDonebaleClass = PipelineManagerDoneable.class)
@Slf4j
public class PipelineManagerController implements ResourceController<PipelineManager> {

  static final String KIND = "PipelineManager";
  static final String GROUP = "sample.javaoperatorsdk";

  private final OpenShiftClient osClient;

  public PipelineManagerController(OpenShiftClient osClient) {
    Contracts.notNull(osClient, "osClient");
    this.osClient = osClient;
  }

  @Override
  public Optional<PipelineManager> createOrUpdateResource(PipelineManager pipelineManager,
      Context<PipelineManager> context) {
    String ns = "mro-operator-test";

    Route route = loadYaml(Route.class, "route.yaml");
    route.getMetadata().setNamespace(ns);

    log.info("Creating or updating Route {} in {}", route.getMetadata().getName(), ns);
    osClient.routes().createOrReplace(route);

    return Optional.of(pipelineManager);
  }

  @Override
  public boolean deleteResource(PipelineManager nginx, Context<PipelineManager> context) {
    return true;
  }

  private <T> T loadYaml(Class<T> clazz, String yaml) {
    try (InputStream is = getClass().getResourceAsStream(yaml)) {
      return Serialization.unmarshal(is, clazz);
    } catch (IOException ex) {
      throw new IllegalStateException("Cannot find yaml on classpath: " + yaml);
    }
  }
}

The implementation of the other classes are identical to your WebServer example, we are passing a working OpenShiftClient to the Controller upon creation (I also tried to just cast the KubernetesClient from the Context class to an OpenShiftClient which leads to the same results). Before the Controller starts the watch, it is possible to create/list things from the OpenShiftApi via the OpenShiftClient. Also creating everything else like ServiceAccount, Service, Role, Rolebindings, Deployment, Jobs, Configmaps is working perfectly as it is all not related to the OpenShift API.

We are currently using OpenShift 4.2.10, with Kubernetes v1.14.6+17b1cc6 and java-operator-sdk 0.3.9.

Any informations on this are greatly appreciated.

Thank you very much.

[csviri]

If everything goes right, this month.

[MroKnowis]

Is there already a date where you aim to relase the v1.0?