Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Using containerd (and not Docker) with BinderHub

See original GitHub issue

Docker is deprecated as a container runtime in Kubernetes 1.20. For those who are switching to a different container runtime, we need a different solution for running repo2docker.

There are some documentation on using Docker-in-Docker, but from what I can tell it doesn’t work with containerd as the container runtime (at least, not with the current helm chart, which mounts dockersocket-host into the dind pod).

I’ve tested on Kubernetes 1.20.0 with containerd 1.4.3 and no Docker, and either the build pod (without DinD) or the DinD pod (with DinD) are not ready because Kubernetes can’t mount /var/run/docker.sock into the pod.

Is there a particular reason why DinD needs access to the host docker.sock? Can DinD run in containerd instead of Docker? I can do some testing if needed.

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:11 (11 by maintainers)

Top GitHub Comments

6reactions

rkevin-archcommented, Dec 13, 2020

I’ve gotten binder to work with dind on top of containerd. Building and spawning images work, and I tested Thebe with it as well.

I did notice because dind and host containerd do not share an image cache, the image has to be pushed to the registry and repulled using the actual container runtime on the same host, which can be a bit slow during the first spawn. Other than that, I don’t feel any differences using this binderhub and one running on top of docker (it’s running at https://binder.galaxy.rkevin.dev, feel free to try it if you want).

I think it’s nice to add the following to the documentation:

If you are using containerd / CRI-O / some container runtime other than Docker, you must do the following:

Enable DinD by setting dind.enabled.
Depending on your network configuration, you should set a MTU smaller than 1500 for the DinD daemon using dind.daemonset.extraArgs. If your apt update during builds just hangs as if the remote server is not responding, definitely double check this. (I’m using Calico as the CNI plugin, and I needed to set a smaller MTU.)
Disable the image cleaner entirely, or set imageCleaner.host.enabled to be false.

Relevant parts of my helm values:

dind:
  enabled: true
  daemonset:
    image:
      name: docker
      tag: 19.03.14-dind
    extraArgs:
      - --mtu
      - "1400"

imageCleaner:
  host:
    enabled: false

3reactions

rkevin-archcommented, Dec 13, 2020

Update 3: The tl;dr for the following is depending on your networking, you should set a smaller MTU for the dind docker daemon. I did the following and it solved the issue:

dind:
  enabled: true
  daemonset:
    image:
      name: docker
      tag: 19.03.14-dind
    extraArgs:
      - --mtu
      - "1400"

See also: this.

Original comment:

Experiencing some really really weird stuff in the build container. I was able to manually run commands in the docker container being built by using kubectl exec to get a shell in the dind container, then using docker -H /run/dind/docker.sock to find the container currently being built and exec into it. Both networking and DNS seem to work properly, which is really weird. This also happened:

root@1adf42268642:/# curl http://archive.ubuntu.com -v
* Rebuilt URL to: http://archive.ubuntu.com/
*   Trying 91.189.88.142...
* TCP_NODELAY set
* Connected to archive.ubuntu.com (91.189.88.142) port 80 (#0)
> GET / HTTP/1.1
> Host: archive.ubuntu.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sun, 13 Dec 2020 09:01:26 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Vary: Accept-Encoding
< Content-Length: 696
< Content-Type: text/html;charset=UTF-8
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
 </head>
 <body>
<h1>Index of /</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th></tr>
   <tr><th colspan="4"><hr></th></tr>
<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="ubuntu/">ubuntu/</a></td><td align="right">2020-12-13 08:25  </td><td align="right">  - </td></tr>
   <tr><th colspan="4"><hr></th></tr>
</table>
<address>Apache/2.4.29 (Ubuntu) Server at archive.ubuntu.com Port 80</address>
</body></html>
* Connection #0 to host archive.ubuntu.com left intact
root@1adf42268642:/# curl -v 91.189.88.142:80
* Rebuilt URL to: 91.189.88.142:80/
*   Trying 91.189.88.142...
* TCP_NODELAY set
* Connected to 91.189.88.142 (91.189.88.142) port 80 (#0)
> GET / HTTP/1.1
> Host: 91.189.88.142
> User-Agent: curl/7.58.0
> Accept: */*
> 
^C

Basically if I curl archive.ubuntu.com, it resolves to 91.189.88.142 and was able to access the webpage. If I curl 91.189.88.142:80 inside the build container, it hangs forever. This makes no sense to me, and I confirmed in the dind pod / host / my own machine that curl -v 91.189.88.142:80 should work as well. It’s only in the currently building Docker container that it has this issue.

The exact output repo2docker gives (after hanging for a very long time):

Step 3/72 : RUN apt-get -qq update &&     apt-get -qq install--yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Running in 1adf42268642
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Connection failed [IP: 91.189.88.142 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Connection failed [IP: 91.189.88.152 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Connection failed [IP: 91.189.88.152 80]
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Connection failed [IP: 91.189.91.39 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
E: Package 'locales' has no installation candidate

I’ll dig around more and see if there’s anything I can find. If we can get dind working in containerd, it would be a great temporary solution until repo2docker stops relying on docker build

Update: curl -v http://archive.ubuntu.com/ubuntu/ hangs in the container being built, but not in the dind container itself. Can replicate using the following Dockerfile:

FROM buildpack-deps:bionic
RUN curl -v http://archive.ubuntu.com/ubuntu/

Meanwhile curl -v http://archive.ubuntu.com/ubuntu and curl -v http://archive.ubuntu.com/ both work. This is very weird.

Update 2: It might be a MTU issue. (By the way, apologies for spamming this thread, I hope other people will find this useful but I’m not sure if this is getting off topic)