Cannot wget/curl/git/apt-get to internet from within docker container

Cannot wget/curl/git/apt-get to internet from within docker container

Reproduction Steps:

1. Create a workspace with custom stack. In the dockerfile, there is a “sudo apt-get update” statement
2. From the console log, I can observe that the request stops at [waiting header]

Expected behavior: Able to do sudo apt-get update and install software.

Observed behavior: I can observe from the console log that the apt-get update stops at [waiting header].

I troubleshoot the issue with the following steps. Confirm that able to ping but when doing wget/curl it seems to be able to connect but not getting response.

ubuntu@dev14-04:~$ docker run -itd --name=container1 busybox
545bf50a46d798602e21d04b177ab9500c48af5c8b37b2773204067631507211
ubuntu@dev14-04:~$ docker attach container1
/ # ping -w3 www.google.com
PING www.google.com (202.75.147.25): 56 data bytes
64 bytes from 202.75.147.25: seq=0 ttl=53 time=24.949 ms
64 bytes from 202.75.147.25: seq=1 ttl=53 time=22.862 ms
64 bytes from 202.75.147.25: seq=2 ttl=53 time=22.751 ms

--- www.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 22.751/23.520/24.949 ms
/ # wget http://www.eclipse.org/che/
Connecting to www.eclipse.org (198.41.30.198:80)

In another container with curl

user@11611da3d377:/projects$ curl -O https://github.com/lhwong/datasciencecoursera/blob/master/HelloWorld.md
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0

I don’t have the same issue when running on Windows.

Che version: 14.5.1 OS and version: Ubuntu 16.04
Docker version: 1.12.0
Che install: Docker container, server (zip)

Additional information:

• Problem started happening recently, didn’t happen in an older version of Che: No
• Problem can be reliably reproduced, doesn’t happen randomly: Yes

The problem persists when firewall is disabled.

In container:

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:03
          inet addr:172.17.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:63 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8423 (8.2 KiB)  TX bytes:5336 (5.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The network bridge seems OK:

ubuntu@dev14-04:~$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "6e86e16d0e75de89442fc4dc672928c35cdd1e66d886f8eacb61beb0361f6006"                                                                                                             ,
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "11611da3d3772e15c528d58170bc515f605b33393e4e506f304841fdc1893d08":                                                                                                              {
                "Name": "ubuntu",
                "EndpointID": "7cf5e799e3a2bade5c8b6632cf30f3c7c499d5a60abf6c946                                                                                                             dd2601bdeec3f7b",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "129d1d5af39a1f5d0a823f91d89ae41b48a2ca240b2190616d36dd75c217cd27":                                                                                                              {
                "Name": "container1",
                "EndpointID": "560a488340a965f158809feccbd7343ebe3a80ddcd2e06419                                                                                                             23171322407b1c4",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "545bf50a46d798602e21d04b177ab9500c48af5c8b37b2773204067631507211":                                                                                                              {
                "Name": "container2",
                "EndpointID": "146d32a022c46daebc5cbf286ae29dac47ca615fcc4a11a2b                                                                                                             b405a0b9317ad70",
                "MacAddress": "02:42:ac:11:00:04",
                "IPv4Address": "172.17.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

[TylerJewell]

@lhwong yes, unfortunately, Eugene is correct here. There is something that is preventing your Docker containers from communicating with the host networking appropriately. Do you know if there are any non-standard docker-engine configuration that has been done to this system? You may want to consider adding --net=host to the way that you launched the container.

But unless we can diagnose the root cause of the docker issues - then che is going to be subject to the same issues.

[lhwong]

iptables and ufw look good. I believe this is issue of connectivity of our network here. Thanks for your help.