Pods stuck in ContainerCreating on AKS due to "MountVolume.SetUp failed for volume "docker-socket" : hostPath type check failed: /var/run/docker.sock is not a socket file"

Bug description

Following the otherwise excellent zero-to-binderhub guide on Azure Kubernetes Service (AKS), the binderhub-image-cleaner and build-pods are stuck in ContainerCreating, causing Binderhub to hault when opening a repo in the frontend

Expected behaviour

Binderhub to start running pods as they are initiated from the frontend

Actual behaviour

Containers are stuck in ContainerCreating.

kubectl get pod

NAME READY STATUS RESTARTS AGE binder-5cd459bb48-dbshg 1/1 Running 0 22m binderhub-image-cleaner-2xwp5 0/1 ContainerCreating 0 26m binderhub-image-cleaner-67jp8 0/1 ContainerCreating 0 26m binderhub-image-cleaner-fngnb 0/1 ContainerCreating 0 26m build-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3-507-ae 0/1 ContainerCreating 0 20m hub-679cbfcdfb-v9vlq 1/1 Running 0 24m proxy-b9cfd6db4-g9djn 1/1 Running 0 24m user-scheduler-7b68bd9d58-4lbvm 1/1 Running 0 26m user-scheduler-7b68bd9d58-szwlc 1/1 Running 0 26m

Doing a

kubectl describe pod

reveals that #810 is not the case here, as all pods are created in the provided namespace.

However, the events reveal the following:

Events: Type Reason Age From Message

---

Normal Scheduled 12m default-scheduler Successfully assigned binderhub/build-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3-507-ae to aks-nodepool1-35493018-vmss000002 Warning FailedMount 3m59s (x4 over 10m) kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-socket docker-config kube-api-access-rngrq]: timed out waiting for the condition Warning FailedMount 102s kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[kube-api-access-rngrq docker-socket docker-config]: timed out waiting for the condition Warning FailedMount 29s (x14 over 12m) kubelet MountVolume.SetUp failed for volume “docker-socket” : hostPath type check failed: /var/run/docker.sock is not a socket file

How to reproduce

1. Set up a fresh resource group / cluster / containerregistry on AKS
2. Follow the instructions on the zero-to-binderhub guide
3. Log on to binder via http://external-ip from kubectl get svc (binder)
4. Observe frontend stuck at “Waiting for build to start…”, and the new build-container stuck in “ContainerCreating” together with three binderhub-image-cleaner containers
5. Do a kubectl describe pod to reveal in the events the following error:

`Warning FailedMount 29s (x14 over 12m) kubelet MountVolume.SetUp failed for volume “docker-socket” : hostPath type check failed: /var/run/docker.sock is not a socket file``

Your personal set up

• Local OS: macOS 11.4, helm version.BuildInfo{Version:“v3.6.2”, GitCommit:“ee407bdf364942bcb8e8c665f82e15aa28009b71”,

• AKS/Binderhub:

AKS (with and without autoscaling) Binderhub 0.2.0-n645.h06139f0

Full outputs:

kubectl describe deployment binder

Name: binder Namespace: binderhub CreationTimestamp: Fri, 06 Aug 2021 09:42:24 +0200 Labels: app.kubernetes.io/managed-by=Helm Annotations: deployment.kubernetes.io/revision: 2 meta.helm.sh/release-name: binderhub meta.helm.sh/release-namespace: binderhub Selector: app=binder,component=binder,release=binderhub Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 0 max unavailable, 1 max surge Pod Template: Labels: app=binder component=binder heritage=Helm name=binder release=binderhub Annotations: checksum/config-map: bf3518883aba64f5e06a30baa56237c2be3505735e23aa8656f2b7d845ccd0e9 checksum/secret: <blankedout> Service Account: binderhub Containers: binder: Image: jupyterhub/k8s-binderhub:0.2.0-n641.h73c6ebd Port: 8585/TCP Host Port: 0/TCP Args: –config /etc/binderhub/config/binderhub_config.py Requests: cpu: 200m memory: 512Mi Liveness: http-get http://:binder/versions delay=10s timeout=10s period=5s #success=1 #failure=3 Readiness: http-get http://:binder/versions delay=0s timeout=3s period=5s #success=1 #failure=1000 Environment: BUILD_NAMESPACE: (v1:metadata.namespace) JUPYTERHUB_API_TOKEN: <set to the key ‘hub.services.binder.apiToken’ in secret ‘hub’> Optional: false Mounts: /etc/binderhub/config/ from config (rw) /etc/binderhub/secret/ from secret-config (rw) /root/.docker from docker-secret (ro) Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: binder-config Optional: false secret-config: Type: Secret (a volume populated by a Secret) SecretName: binder-secret Optional: false docker-secret: Type: Secret (a volume populated by a Secret) SecretName: binder-build-docker-config Optional: false Conditions: Type Status Reason

---

Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: binder-5cd459bb48 (1/1 replicas created) Events: Type Reason Age From Message

---

Normal ScalingReplicaSet 34m deployment-controller Scaled up replica set binder-5f8d69cf76 to 1 Normal ScalingReplicaSet 30m deployment-controller Scaled up replica set binder-5cd459bb48 to 1 Normal ScalingReplicaSet 30m deployment-controller Scaled down replica set binder-5f8d69cf76 to 0

kubectl describe pod build-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3-507-ae

Name: build-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3-507-ae Namespace: binderhub Priority: 0 Node: aks-nodepool1-35493018-vmss000002/10.240.0.66 Start Time: Fri, 06 Aug 2021 09:48:06 +0200 Labels: component=binderhub-build name=build-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3-507-ae Annotations: binder-repo: https://github.com/bjornarfjelldal/binderhubtest_public Status: Pending IP:
IPs: <none> Containers: builder: Container ID:
Image: jupyter/repo2docker:2021.01.0 Image ID:
Port: <none> Host Port: <none> Args: jupyter-repo2docker –ref 507897685c11f5c0984d097c541b614edf12c6d2 –image shopacrbinderhub.azurecr.io/binderhub/binder-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3:507897685c11f5c0984d097c541b614edf12c6d2 –no-clean –no-run –json-logs –user-name jovyan –user-id 1000 –push https://github.com/bjornarfjelldal/binderhubtest_public State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 0 Requests: memory: 0 Environment: <none> Mounts: /root/.docker from docker-config (rw) /var/run/docker.sock from docker-socket (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rngrq (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: docker-socket: Type: HostPath (bare host directory volume) Path: /var/run/docker.sock HostPathType: Socket docker-config: Type: Secret (a volume populated by a Secret) SecretName: binder-build-docker-config Optional: false kube-api-access-rngrq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: hub.jupyter.org/dedicated=user:NoSchedule hub.jupyter.org_dedicated=user:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Normal Scheduled 28m default-scheduler Successfully assigned binderhub/build-bjornarfjelldal-2dbinderhubtest-5fpublic-ea7ba3-507-ae to aks-nodepool1-35493018-vmss000002 Warning FailedMount 14m (x2 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[kube-api-access-rngrq docker-socket docker-config]: timed out waiting for the condition Warning FailedMount 12m kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-config kube-api-access-rngrq docker-socket]: timed out waiting for the condition Warning FailedMount 5m37s (x7 over 26m) kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-socket docker-config kube-api-access-rngrq]: timed out waiting for the condition Warning FailedMount 91s (x21 over 28m) kubelet MountVolume.SetUp failed for volume “docker-socket” : hostPath type check failed: /var/run/docker.sock is not a socket file

Let me know if anything else is needed!

Edit: input files, obviously. helm upgrade --cleanup-on-fail \ --install binderhub jupyterhub/binderhub \ --namespace binderhub \ --create-namespace \ --version=0.2.0-n645.h06139f0 \ --values config.yaml \ --values secret.yaml is ran with

config.yaml:

config: BinderHub: hub_url: http://external-ip-from-binder-pod use_registry: true image_prefix: shopacrbinderhub.azurecr.io/binderhub/binder- DockerRegistry: token_url: “https://shopacrbinderhub.azurecr.io/oauth2/token?service=shopacrbinderhub.azurecr.io”

secret.yaml:

registry: url: https://shopacrbinderhub.azurecr.io username: <appId> password: <password>

[mattchildsupp]

It never ceases to amaze me how you try solving something for days, countless hours searching, but the factor solving it is asking for help, before suddenly figuring something out.

I needed to enable dind:

dind: enabled: true daemonset: image: name: docker tag: 18.09.2-dind

A setting I missed as it was at the very botttom of the tutorial, and it didn’t seem like something obligatory. Sorry for the inconvenience again, @consideRatio, but thank you for your response!

Would it be possible for you to share a code snippet of your entire config file that you got working in the end?

[taghillbilly]

“Would it be possible for you to share a code snippet of your entire config file that you got working in the end?” I also second this. For me, the symptom is exactly the same. But applying the ‘dind’ syntax to config.yaml and then upgrade the helm does not help. So, my guess is the syntax is still wrong. Should there be indent in front of dind?