The workspace agent has attempted to start, but it is unable to ping the Che server at http://che-host:8080/ide/api/

Description of problem:

• The workspace agent has attempted to start, but it is unable to ping the Che server at http://che-host:8080/ide/api/

Che version:

• 4.0.0-RC13

Che install directory:

• C:\Users\Eric\AppData\Local\Programs\Eclipse-Che\eclipse-che-4.0.0-RC13

Docker version:

• Client:
• Version: 1.9.1
• API version: 1.21
• Go version: go1.4.3
• Git commit: a34a1d5
• Built: Fri Nov 20 17:56:04 UTC 2015
• OS/Arch: windows/amd64
• An error occurred trying to connect: Get http://localhost:2375/v1.21/version: dial tcp 127.0.0.1:2375: ConnectEx tcp: No connection could be made because the target machine actively refused it.

Java version:

• java version “1.8.0_73”
• Java™ SE Runtime Environment (build 1.8.0_73-b02)
• Java HotSpot™ 64-Bit Server VM (build 25.73-b02, mixed mode)

Environment details (AWS, VirtualBox, physical, etc.):

• Windows 10

How reproducible:

• Always

Steps to Reproduce:

1. Create new work-space
2. Run new work-space
3. Inspect output from http://localhost:8080/dashboard/#/ide/MyFirstWorkspace

Actual Results:

[STDOUT] Listening for transport dt_socket at address: 4403
[STDOUT] 2016-03-10 23:35:00,806[main]             [INFO ] [o.a.c.s.VersionLoggerListener 89]    - Server version:        Apache Tomcat/8.0.29
[STDOUT] 2016-03-10 23:35:00,814[main]             [INFO ] [o.a.c.s.VersionLoggerListener 91]    - Server built:          Nov 20 2015 09:18:00 UTC
[STDOUT] 2016-03-10 23:35:00,820[main]             [INFO ] [o.a.c.s.VersionLoggerListener 93]    - Server number:         8.0.29.0
[STDOUT] 2016-03-10 23:35:00,834[main]             [INFO ] [o.a.c.s.VersionLoggerListener 95]    - OS Name:               Linux
[STDOUT] 2016-03-10 23:35:00,834[main]             [INFO ] [o.a.c.s.VersionLoggerListener 97]    - OS Version:            4.1.18-boot2docker
[STDOUT] 2016-03-10 23:35:00,835[main]             [INFO ] [o.a.c.s.VersionLoggerListener 99]    - Architecture:          amd64
[STDOUT] 2016-03-10 23:35:00,835[main]             [INFO ] [o.a.c.s.VersionLoggerListener 101]   - Java Home:             /opt/jdk1.8.0_45/jre
[STDOUT] 2016-03-10 23:35:00,835[main]             [INFO ] [o.a.c.s.VersionLoggerListener 103]   - JVM Version:           1.8.0_45-b14
[STDOUT] 2016-03-10 23:35:00,836[main]             [INFO ] [o.a.c.s.VersionLoggerListener 105]   - JVM Vendor:            Oracle Corporation
[STDOUT] 2016-03-10 23:35:00,837[main]             [INFO ] [o.a.c.s.VersionLoggerListener 107]   - CATALINA_BASE:         /home/user/che/ws-agent
[STDOUT] 2016-03-10 23:35:00,837[main]             [INFO ] [o.a.c.s.VersionLoggerListener 109]   - CATALINA_HOME:         /home/user/che/ws-agent
[STDOUT] 2016-03-10 23:35:00,838[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.config.file=/home/user/che/ws-agent/conf/logging.properties
[STDOUT] 2016-03-10 23:35:00,839[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
[STDOUT] 2016-03-10 23:35:00,842[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Xms256m
[STDOUT] 2016-03-10 23:35:00,843[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Xmx1024m
[STDOUT] 2016-03-10 23:35:00,846[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.security.egd=file:/dev/./urandom
[STDOUT] 2016-03-10 23:35:00,847[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.dir=/home/user/che/ws-agent/logs
[STDOUT] 2016-03-10 23:35:00,849[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote
[STDOUT] 2016-03-10 23:35:00,850[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.ssl=false
[STDOUT] 2016-03-10 23:35:00,850[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
[STDOUT] 2016-03-10 23:35:00,851[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.local.conf.dir=/mnt/che/conf
[STDOUT] 2016-03-10 23:35:00,862[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -agentlib:jdwp=transport=dt_socket,address=4403,server=y,suspend=n
[STDOUT] 2016-03-10 23:35:00,863[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.endorsed.dirs=/home/user/che/ws-agent/endorsed
[STDOUT] 2016-03-10 23:35:00,864[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.base=/home/user/che/ws-agent
[STDOUT] 2016-03-10 23:35:00,871[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.home=/home/user/che/ws-agent
[STDOUT] 2016-03-10 23:35:00,871[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.io.tmpdir=/home/user/che/ws-agent/temp
[STDOUT] 2016-03-10 23:35:01,208[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 425]  - Initializing ProtocolHandler ["http-nio-4401"]
[STDOUT] 2016-03-10 23:35:01,237[main]             [INFO ] [o.a.t.util.net.NioSelectorPool 72]   - Using a shared selector for servlet write/read
[STDOUT] 2016-03-10 23:35:01,248[main]             [INFO ] [o.a.catalina.startup.Catalina 587]   - Initialization processed in 1172 ms
[STDOUT] 2016-03-10 23:35:01,304[main]             [INFO ] [c.m.JmxRemoteLifecycleListener 332]  - The JMX Remote Listener has configured the registry on port 32002 and the server on port 32102 for the Platform server
[STDOUT] 2016-03-10 23:35:01,305[main]             [INFO ] [o.a.c.core.StandardService 433]      - Starting service Catalina
[STDOUT] 2016-03-10 23:35:01,310[main]             [INFO ] [o.a.c.core.StandardEngine 259]       - Starting Servlet Engine: Apache Tomcat/8.0.29
[STDOUT] 2016-03-10 23:35:01,447[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 916]        - Deploying web application archive /home/user/che/ws-agent/webapps/ide.war
[STDOUT] Thu Mar 10 23:35:13 UTC 2016 - [localhost-startStop-1] Product-specified preferences called before plugin is started
[STDOUT] 2016-03-10 23:35:15,482[ost-startStop-1]  [ERROR] [piEndpointAccessibilityChecker 62]   - connect timed out
[STDOUT] java.net.SocketTimeoutException: connect timed out
[STDOUT]    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_45]
[STDOUT]    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) ~[na:1.8.0_45]
[STDOUT]    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_45]
[STDOUT]    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_45]
[STDOUT]    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_45]
[STDOUT]    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_45]
[STDOUT]    at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512) ~[na:1.8.0_45]
[STDOUT]    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) ~[na:1.8.0_45]
[STDOUT]    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_45]
[STDOUT]    at org.eclipse.che.api.core.rest.DefaultHttpJsonRequest.doRequest(DefaultHttpJsonRequest.java:239) ~[che-core-api-core-4.0.0-RC13.jar:4.0.0-RC13]
[STDOUT]    at org.eclipse.che.api.core.rest.DefaultHttpJsonRequest.request(DefaultHttpJsonRequest.java:145) ~[che-core-api-core-4.0.0-RC13.jar:4.0.0-RC13]
[STDOUT]    at org.eclipse.che.wsagent.server.ApiEndpointAccessibilityChecker.start(ApiEndpointAccessibilityChecker.java:57) ~[classes/:4.0.0-RC13]
[STDOUT]    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
[STDOUT]    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
[STDOUT]    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
[STDOUT]    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
[STDOUT]    at org.eclipse.che.inject.lifecycle.InitModule$1$1.afterInjection(InitModule.java:44) [che-core-commons-inject-4.0.0-RC13.jar:4.0.0-RC13]
[STDOUT]    at com.google.inject.internal.MembersInjectorImpl.notifyListeners(MembersInjectorImpl.java:119) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:115) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) [guice-4.0.jar:na]
[STDOUT]    at com.google.inject.Guice.createInjector(Guice.java:96) [guice-4.0.jar:na]
[STDOUT]    at org.everrest.guice.servlet.EverrestGuiceContextListener.getInjector(EverrestGuiceContextListener.java:137) [everrest-integration-guice-1.12.1.jar:na]
[STDOUT]    at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:47) [guice-servlet-4.0.jar:na]
[STDOUT]    at org.everrest.guice.servlet.EverrestGuiceContextListener.contextInitialized(EverrestGuiceContextListener.java:84) [everrest-integration-guice-1.12.1.jar:na]
[STDOUT]    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4793) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5236) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:945) [catalina.jar:8.0.29]
[STDOUT]    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1795) [catalina.jar:8.0.29]
[STDOUT]    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
[STDOUT]    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
[STDOUT]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
[STDOUT]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
[STDOUT]    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[STDOUT] 2016-03-10 23:35:15,494[ost-startStop-1]  [ERROR] [piEndpointAccessibilityChecker 65]   - The workspace agent has attempted to start, but it is unable to ping the Che server at http://che-host:8080/ide/api/
[STDOUT] 2016-03-10 23:35:15,496[ost-startStop-1]  [ERROR] [piEndpointAccessibilityChecker 66]   - The workspace agent has been forcefully stopped. This error happens when the agent cannot resolve the location of the Che server. This error can usually be fixed with additional configuration settings in /conf/che.properties. The Che server will stop this workspace after a short timeout. You can get help by posting your config, stacktrace and workspace /etc/hosts below as a GitHub issue.
[STDOUT] 2016-03-10 23:35:15,497[ost-startStop-1]  [INFO ] [piEndpointAccessibilityChecker 74]   - Workspace /etc/hosts: 127.0.0.1  localhost
[STDOUT] ::1    localhost ip6-localhost ip6-loopback
[STDOUT] fe00::0    ip6-localnet
[STDOUT] ff00::0    ip6-mcastprefix
[STDOUT] ff02::1    ip6-allnodes
[STDOUT] ff02::2    ip6-allrouters
[STDOUT] 192.168.99.1   che-host
[STDOUT] 172.17.0.2 32c45483c75c
[STDOUT] 
[ERROR] Timeout reached. The Che server has been unable to verify that your workspace's agent has successfully booted. Either the workspace is unreachable, the agent had an error during startup, or your workspace is starting slowly. You can configure machine.ws_agent.max_start_time_ms in Che properties to increase the timeout.

Observations:

• I can ping http://localhost:8080/ide/api/
• {“rootResources”:[{“path”:“project-template”,“regex”:“/project-template(/.)?“,“fqn”:“org.eclipse.che.api.project.server.ProjectTemplateService”},{“path”:“event-bus”,“regex”:”/event-bus(/.)?”,“fqn”:“org.eclipse.che.api.core.notification.WSocketEventBusServer”},{“path”:“/workspace”,“regex”:“/workspace(/.)?“,“fqn”:“org.eclipse.che.api.workspace.server.WorkspaceService”},{“path”:”/machine",“regex”:"/machine(/.)?”,“fqn”:“org.eclipse.che.api.machine.server.MachineService”},{“path”:“/profile”,“regex”:“/profile(/.)?“,“fqn”:“org.eclipse.che.api.user.server.UserProfileService”},{“path”:”/async/{ws-id}“,“regex”:”/async/([^/]+?)(/.)?”,“fqn”:“org.everrest.core.impl.async.AsynchronousJobService”},{“path”:“/recipe”,“regex”:“/recipe(/.)?“,“fqn”:“org.eclipse.che.api.machine.server.recipe.RecipeService”},{“path”:“oauth”,“regex”:”/oauth(/.)?”,“fqn”:“org.eclipse.che.security.oauth.OAuthAuthenticationService”},{“path”:“/stack”,“regex”:“/stack(/.)?“,“fqn”:“org.eclipse.che.api.workspace.server.stack.StackService”},{“path”:”/auth",“regex”:"/auth(/.)?”,“fqn”:“org.eclipse.che.api.auth.AuthenticationService”},{“path”:“/docs”,“regex”:“/docs(/.)?“,“fqn”:“org.eclipse.che.swagger.rest.SwaggerSpecificationService”},{“path”:”/user",“regex”:"/user(/.)?”,“fqn”:“org.eclipse.che.api.user.server.UserService”},{“path”:“/ssh”,“regex”:“/ssh(/.)?“,“fqn”:“org.eclipse.che.api.ssh.server.SshService”},{“path”:”/“,“regex”:”(/.)?”,“fqn”:“org.eclipse.che.api.core.rest.ApiInfoService”}]}
• Could it really be looking for che-host:8080?
• I tried creating C:\Users\Eric\AppData\Local\Eclipse Che\vfs\MyFirstWorkspace\etc\hosts with che-host, but that does not seem to work

[TylerJewell]

Hi @kolotyluk. I believe this is a netowrking configuration issue that you are running into. Any chance that you are running Che on a server that has different networks and IP addresses? It looks like Che is having a hard time resolving certain addresses.

You can see more about certain non-standard configurations here. https://eclipse-che.readme.io/docs/networking

You wrote a great issue by the way. Very clear and provides all the background information.

[TylerJewell]

@pavan5000 Hey - I am really happy to see this working for you. I am going to close the issue.