question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Compatibility with containerd?

See original GitHub issue

I use Bottlerocket AMI on EKS clusters which use containerd that does not uses docker or docker socket.

Custom actions fails with errors like these. As a workaround, is there a way I can use a pre-pulled docker image instead of GitHub action trying to build an image on the fly?

Build container for action use: '/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile'.
  /usr/local/bin/docker build -t 60e226:7de2787af7e04b038ce49eb6a1a987d8 -f "/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile" "/runner/_work/_actions/aevea/commitsar/v0.16.0"
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  Warning: Docker build failed with exit code 1, back off 8.072 seconds before retry.
  /usr/local/bin/docker build -t 60e226:7de2787af7e04b038ce49eb6a1a987d8 -f "/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile" "/runner/_work/_actions/aevea/commitsar/v0.16.0"
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  Warning: Docker build failed with exit code 1, back off 4.07 seconds before retry.
  /usr/local/bin/docker build -t 60e226:7de2787af7e04b038ce49eb6a1a987d8 -f "/runner/_work/_actions/aevea/commitsar/v0.16.0/Dockerfile" "/runner/_work/_actions/aevea/commitsar/v0.16.0"
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Error: Docker build failed with exit code 1

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:22
  • Comments:7 (1 by maintainers)

github_iconTop GitHub Comments

7reactions
jmorcarcommented, Aug 12, 2022

Any update about this? Images with Docker in docker is a segurity risk and today docker is deprecated in Cloud environment with kubernetes, so docker.sock won’t found any more.

Here an issue related with the alternative (deploy docker cli without docker daemon instance)… conclusion it docker always need a daemon started with its docker.sock, so this confirm there is not alternative to use docker in docker to pull images in nodes migrated with new dockershim like contairnerd

5reactions
corrigatcommented, Apr 20, 2023

Chiming in as an affected consumer, with EKS dropping K8S 1.22 in June. I’m reading what may be some misunderstandings of the capabilities of containerd and the desire to use docker in runner pods. If runner pods were able to call docker and launch containers before, they will still be able to after upgrading to the latest K8S version. At least they will if you’re using the summerwind-dind container or built your own and borrowed the pieces that install dockerd and supervisord. The ARC dind containers are already launched with the privileged security context, so dockerd will still work so long as the node has docker engine installed, and the runnerdeployment mounts the socket and /var/lib/docker into the runner.

kubectl version --short
Working in namespace default!
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.25.6-eks-48e63af
...
kubectl exec -it corrigat-testing -- /bin/bash
builder@corrigat-testing:/$ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
...
builder@corrigat-testing:/$ docker run alpine:latest echo hello
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
63b65145d645: Pull complete
Digest: sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517
Status: Downloaded newer image for alpine:latest
hello

The difference in K8S without dockershim is we can no longer share the imagefs directory or runtime socket with the runner pods. While this does not totally affect the ability to process workflow jobs, and even workflows building or using containers with docker will continue to work, without containerd support the affect is image pulls in new and ephemeral runners will always require a full image fetch from the registry. It has been a great help to mount the docker socket and imagefs into runner pods - we have processed 2,325,917 (accurate as of now) workflow runs with ARC, and saved at least 2/3 of that times our 3GB image size in image pulls and the associated time. It would be fantastic to be able to continue that with containerd. Additionally, imagefs storage sizing will need to be reconsidered to facilitate the additional copies of container images stored locally in each pod’s overlayfs on worker nodes.

I’ve played with just continuing to install docker in our worker node AMI, but Kubelet doesn’t try to manage anything docker related under disk pressure, so the node just fills up and dies.

Looking forward to updates here.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Versioning and release
x patch releases are always backward compatible with its major and minor version. Minor (0.x.0) version will always be compatible with the previous...
Read more >
Releases · containerd/containerd
The Node Resource Interface is a common framework for plugging extensions into OCI-compatible container runtimes. It provides
Read more >
Container Runtimes
Your container runtime must support at least v1alpha2 of the container runtime interface. Kubernetes starting v1.26 only works with v1 of the ...
Read more >
Containerd for your Compatibility | by Muhammad Usama
Containerd is the most stable, best supported runtime for Kubernetes and most major cloud providers. For development, Docker and buildkit are highly integrated ......
Read more >
Nerdctl: Docker compatible containerd command line tool
Nerdctl works in tandem with containerd, serving as a compatible cli for containerd and offering support for many docker cli commands.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found