Using containerd (and not Docker) with BinderHubSee original GitHub issue
Docker is deprecated as a container runtime in Kubernetes 1.20. For those who are switching to a different container runtime, we need a different solution for running repo2docker.
There are some documentation on using Docker-in-Docker, but from what I can tell it doesn’t work with containerd as the container runtime (at least, not with the current helm chart, which mounts
dockersocket-host into the dind pod).
I’ve tested on Kubernetes 1.20.0 with containerd 1.4.3 and no Docker, and either the build pod (without DinD) or the DinD pod (with DinD) are not ready because Kubernetes can’t mount
/var/run/docker.sock into the pod.
Is there a particular reason why DinD needs access to the host
docker.sock? Can DinD run in containerd instead of Docker? I can do some testing if needed.
- Created 3 years ago
- Comments:11 (11 by maintainers)
Top GitHub Comments
I’ve gotten binder to work with dind on top of containerd. Building and spawning images work, and I tested Thebe with it as well.
I did notice because dind and host containerd do not share an image cache, the image has to be pushed to the registry and repulled using the actual container runtime on the same host, which can be a bit slow during the first spawn. Other than that, I don’t feel any differences using this binderhub and one running on top of docker (it’s running at https://binder.galaxy.rkevin.dev, feel free to try it if you want).
I think it’s nice to add the following to the documentation:
If you are using containerd / CRI-O / some container runtime other than Docker, you must do the following:
- Enable DinD by setting
- Depending on your network configuration, you should set a MTU smaller than 1500 for the DinD daemon using
dind.daemonset.extraArgs. If your
apt updateduring builds just hangs as if the remote server is not responding, definitely double check this. (I’m using Calico as the CNI plugin, and I needed to set a smaller MTU.)
- Disable the image cleaner entirely, or set
imageCleaner.host.enabledto be false.
Relevant parts of my helm values:
dind: enabled: true daemonset: image: name: docker tag: 19.03.14-dind extraArgs: - --mtu - "1400" imageCleaner: host: enabled: false
Update 3: The tl;dr for the following is depending on your networking, you should set a smaller MTU for the dind docker daemon. I did the following and it solved the issue:
dind: enabled: true daemonset: image: name: docker tag: 19.03.14-dind extraArgs: - --mtu - "1400"
See also: this.
Experiencing some really really weird stuff in the build container. I was able to manually run commands in the docker container being built by using
kubectl exec to get a shell in the
dind container, then using
docker -H /run/dind/docker.sock to find the container currently being built and exec into it. Both networking and DNS seem to work properly, which is really weird. This also happened:
root@1adf42268642:/# curl http://archive.ubuntu.com -v * Rebuilt URL to: http://archive.ubuntu.com/ * Trying 184.108.40.206... * TCP_NODELAY set * Connected to archive.ubuntu.com (220.127.116.11) port 80 (#0) > GET / HTTP/1.1 > Host: archive.ubuntu.com > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 200 OK < Date: Sun, 13 Dec 2020 09:01:26 GMT < Server: Apache/2.4.29 (Ubuntu) < Vary: Accept-Encoding < Content-Length: 696 < Content-Type: text/html;charset=UTF-8 < <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> <html> <head> <title>Index of /</title> </head> <body> <h1>Index of /</h1> <table> <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th></tr> <tr><th colspan="4"><hr></th></tr> <tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="ubuntu/">ubuntu/</a></td><td align="right">2020-12-13 08:25 </td><td align="right"> - </td></tr> <tr><th colspan="4"><hr></th></tr> </table> <address>Apache/2.4.29 (Ubuntu) Server at archive.ubuntu.com Port 80</address> </body></html> * Connection #0 to host archive.ubuntu.com left intact root@1adf42268642:/# curl -v 18.104.22.168:80 * Rebuilt URL to: 22.214.171.124:80/ * Trying 126.96.36.199... * TCP_NODELAY set * Connected to 188.8.131.52 (184.108.40.206) port 80 (#0) > GET / HTTP/1.1 > Host: 220.127.116.11 > User-Agent: curl/7.58.0 > Accept: */* > ^C
Basically if I curl
archive.ubuntu.com, it resolves to
18.104.22.168 and was able to access the webpage. If I curl
22.214.171.124:80 inside the build container, it hangs forever. This makes no sense to me, and I confirmed in the dind pod / host / my own machine that
curl -v 126.96.36.199:80 should work as well. It’s only in the currently building Docker container that it has this issue.
The exact output repo2docker gives (after hanging for a very long time):
Step 3/72 : RUN apt-get -qq update && apt-get -qq install--yes --no-install-recommends locales > /dev/null && apt-get -qq purge && apt-get -qq clean && rm -rf /var/lib/apt/lists/* ---> Running in 1adf42268642 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Connection failed [IP: 188.8.131.52 80] W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Connection failed [IP: 184.108.40.206 80] W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Connection failed [IP: 220.127.116.11 80] W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Connection failed [IP: 18.104.22.168 80] W: Some index files failed to download. They have been ignored, or old ones used instead. E: Package 'locales' has no installation candidate
I’ll dig around more and see if there’s anything I can find. If we can get dind working in containerd, it would be a great temporary solution until repo2docker stops relying on
curl -v http://archive.ubuntu.com/ubuntu/ hangs in the container being built, but not in the dind container itself. Can replicate using the following Dockerfile:
FROM buildpack-deps:bionic RUN curl -v http://archive.ubuntu.com/ubuntu/
curl -v http://archive.ubuntu.com/ubuntu and
curl -v http://archive.ubuntu.com/ both work. This is very weird.
Update 2: It might be a MTU issue. (By the way, apologies for spamming this thread, I hope other people will find this useful but I’m not sure if this is getting off topic)