Workspace start failed!
See original GitHub issueDescribe the bug
We deploy Che on Aliyun K8S,and starts the workspace,java or python all failed!!
Che version
we run docker pull eclipse/che-server:latest Actually che-server run info is 7.7.0, interesting?
Steps to reproduce
Expected behavior
Runtime
-
kubernetes (include output of
kubectl version
) Client Version: version.Info{Major:“1”, Minor:“18”, GitVersion:“v1.18.3”, GitCommit:“2e7996e3e2712684bc73f0dec0200d64eec7fe40”, GitTreeState:“clean”, BuildDate:“2020-05-20T12:52:00Z”, GoVersion:“go1.13.9”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“16+”, GitVersion:“v1.16.9-aliyun.1”, GitCommit:“4f7ea78”, GitTreeState:“”, BuildDate:“2020-05-08T07:29:59Z”, GoVersion:“go1.13.9”, Compiler:“gc”, Platform:“linux/amd64”} -
docker version
Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:22:05 2019 OS/Arch: linux/amd64 Experimental: false
Screenshots
Installation method
helm init && helm dep update helm upgrade --install che --namespace che --set cheImage=eclipse/che-server:latest --set global.cheWorkspacesNamespace=“che” --set global.ingressDomain=172.26.1.208.nip.io ./
Environment
Aliyun
Eclipse Che Logs
Using embedded assembly.
Listening for transport dt_socket at address: 8000
14-Jul-2020 07:01:10.517 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/8.5.46
14-Jul-2020 07:01:10.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Sep 16 2019 18:16:19 UTC
14-Jul-2020 07:01:10.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.46.0
14-Jul-2020 07:01:10.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 3.10.0-1062.9.1.el7.x86_64
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/local/openjdk-8
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_232-b09
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /home/user/eclipse-che/tomcat
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /home/user/eclipse-che/tomcat
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMFraction=2
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MinHeapFreeRatio=10
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxHeapFreeRatio=20
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:GCTimeRatio=4
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:AdaptiveSizePolicyWeight=90
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UnlockExperimentalVMOptions
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseCGroupMemoryLimitForHeap
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dsun.zip.disableMemoryMapping=true
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms20m
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che/
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/data/logs
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/etc/conf
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
14-Jul-2020 07:01:10.604 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
14-Jul-2020 07:01:10.611 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
14-Jul-2020 07:01:10.624 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 520 ms
14-Jul-2020 07:01:10.632 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
14-Jul-2020 07:01:10.632 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.46
14-Jul-2020 07:01:10.651 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
14-Jul-2020 07:01:10.899 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [248] ms
14-Jul-2020 07:01:10.903 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2020-07-14 07:01:14,712[ost-startStop-1] [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63] - This=org.eclipse.che.core.db.h2.H2SQLJndiDataSourceFactory@1d6d44fb obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.api.CommonJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@39cfdee5 environment={}
2020-07-14 07:01:17,658[ost-startStop-1] [INFO ] [o.f.c.i.d.DbSupportFactory 44] - Database: jdbc:h2:che (H2 1.4)
2020-07-14 07:01:17,710[ost-startStop-1] [INFO ] [o.f.c.i.util.VersionPrinter 44] - Flyway 4.2.0 by Boxfuse
2020-07-14 07:01:17,711[ost-startStop-1] [INFO ] [o.f.c.i.d.DbSupportFactory 44] - Database: jdbc:h2:che (H2 1.4)
2020-07-14 07:01:17,740[ost-startStop-1] [INFO ] [i.f.CustomSqlMigrationResolver 158] - Searching for sql scripts in locations [classpath:che-schema]
2020-07-14 07:01:17,794[ost-startStop-1] [INFO ] [o.f.c.i.command.DbValidate 44] - Successfully validated 55 migrations (execution time 00:00.054s)
2020-07-14 07:01:17,809[ost-startStop-1] [INFO ] [o.f.c.i.command.DbMigrate 44] - Current version of schema "PUBLIC": 7.6.0.1
2020-07-14 07:01:17,811[ost-startStop-1] [INFO ] [o.f.c.i.command.DbMigrate 44] - Schema "PUBLIC" is up to date. No migration necessary.
2020-07-14 07:01:19,359[ost-startStop-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 174] - Configured factories for environments: '[kubernetes, no-environment]'
2020-07-14 07:01:19,359[ost-startStop-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 175] - Registered infrastructure 'kubernetes'
2020-07-14 07:01:19,405[ost-startStop-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 651] - Infrastructure is tracking 0 active runtimes
2020-07-14 07:01:19,423[ost-startStop-1] [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36] - Eclipse Che Api Core: Build info '7.7.0' scmRevision '20c990bf70971f956ebe174da2d67f09d6ff9432' implementationVersion '7.7.0'
14-Jul-2020 07:01:19.616 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [8,713] ms
14-Jul-2020 07:01:19.617 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
14-Jul-2020 07:01:19.734 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [116] ms
14-Jul-2020 07:01:19.736 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
14-Jul-2020 07:01:19.923 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Jul-2020 07:01:19.930 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [194] ms
14-Jul-2020 07:01:19.931 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
14-Jul-2020 07:01:19.940 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [9] ms
14-Jul-2020 07:01:19.942 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
14-Jul-2020 07:01:21.319 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [1,377] ms
14-Jul-2020 07:01:21.322 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
14-Jul-2020 07:01:21.330 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 10705 ms
2020-07-14 07:02:10,509[nio-8080-exec-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 469] - Starting workspace 'che/wksp-77xl' with id 'workspaceeq8elhr6jxecraue' by user 'che'
2020-07-14 07:02:18,269[/172.26.0.1/...] [ERROR] [k.c.d.i.WatchConnectionManager 268] - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-07-14 07:02:18,280[/172.26.0.1/...] [ERROR] [k.c.d.i.WatchConnectionManager 268] - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
Additional context
More information
che container Log:
2020-07-14 08:10:16,502[/172.26.0.1/...] [ERROR] [k.c.d.i.WatchConnectionManager 268] - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-07-14 08:10:16,503[/172.26.0.1/...] [ERROR] [k.c.d.i.WatchConnectionManager 268] - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-07-14 08:13:16,604[aceSharedPool-4] [WARN ] [.i.k.KubernetesInternalRuntime 252] - Failed to start Kubernetes runtime of workspace workspaceeq8elhr6jxecraue. Cause: Plugins installation process timed out
2020-07-14 08:13:16,632[aceSharedPool-4] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 897] - Workspace 'che:wksp-77xl' with id 'workspaceeq8elhr6jxecraue' start failed
workspace page:
Issue Analytics
- State:
- Created 3 years ago
- Comments:17 (10 by maintainers)
Top GitHub Comments
TKS, We already make it.
@strongit So, you have an issue with the
Certifacte Error
page, correct? Did you follow this guide to import certificate into a browse? https://www.eclipse.org/che/docs/che-7/installing-che-in-tls-mode-with-self-signed-certificates/#importing-self-signed-tls-certificates-to-browsers_installing-che-in-tls-mode-with-self-signed-certificates Which browse do you use?