Containers stuck at ContainerCreating

I have set up a BinderHub using a modified version of the instructions at https://zero-to-jupyterhub.readthedocs.io. I can load up the main page, but the process does not advance beyond the “Waiting” status, and the log there just shows “Waiting for build to start…”, even after waiting more than an hour.

Looking at the pods, I see a container whose name starts with build-binder, which has been stuck at the ContainerCreating status. Strangely, it appears in the default namespace, even though everything else (BinderHub, JupyterHub) is under a different namespace (sheff-hub).

I can’t access its log, but describe shows that mounting a volume has failed, because of a non-existent secret:

MountVolume.SetUp failed for volume "docker-push-secret" : secrets "binder-push-secret" not found

(full output attached at end)

Indeed, that secret does not exist in the default namespace, but it does in sheff-hub:

NAMESPACE     NAME                                  TYPE                                  DATA   AGE
default       default-token-2fq72                   kubernetes.io/service-account-token   3      4h
sheff-hub     binder-push-secret                    Opaque                                1      2h
sheff-hub     binder-secret                         Opaque                                2      2h
sheff-hub     binderhub-token-wkfzx                 kubernetes.io/service-account-token   3      2h
sheff-hub     default-token-zcdbw                   kubernetes.io/service-account-token   3      4h
sheff-hub     hub-secret                            Opaque                                2      2h
sheff-hub     hub-token-6dd7s                       kubernetes.io/service-account-token   3      2h
sheff-hub     sheff-hub-image-cleaner-token-kfmh5   kubernetes.io/service-account-token   3      2h

(table snipped to remove secrets from kube-system, kube-public namespaces)

This is on OS X, with

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:35:19Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8", GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean", BuildDate:"2019-02-28T18:40:05Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

I also had to use the workaround in #809, because I was encountering the same error when trying to install binderhub, but am not getting the authentication error now.

Thanks in advice for any insight or advice!

---

This is the full output of describing the container:

$ kubectl describe pod build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Name:               build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-nodepool1-33022287-1/10.240.0.6
Start Time:         Mon, 18 Mar 2019 15:03:56 +0000
Labels:             component=binderhub-build
                    name=build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Annotations:        binder-repo: https://github.com/binder-examples/requirements
Status:             Pending
IP:                 
Containers:
  builder:
    Container ID:  
    Image:         jupyter/repo2docker:0.8.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      jupyter-repo2docker
      --ref
      a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
      --image
      binder-2dexamples-2drequirements-55ab5c:a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
      --no-clean
      --no-run
      --json-logs
      --user-name
      jovyan
      --user-id
      1000
      --push
      https://github.com/binder-examples/requirements
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  0
    Requests:
      memory:  0
    Environment:
      KUBERNETES_PORT_443_TCP_ADDR:  sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
      KUBERNETES_PORT:               tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:       tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:       sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
    Mounts:
      /root/.docker from docker-push-secret (rw)
      /var/run/docker.sock from docker-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2fq72 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  docker-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  docker-push-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  binder-push-secret
    Optional:    false
  default-token-2fq72:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-2fq72
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                   From                               Message
  ----     ------       ----                  ----                               -------
  Normal   Scheduled    83m                   default-scheduler                  Successfully assigned default/build-binder-2dexamples-2drequirements-55ab5c-a73ba1 to aks-nodepool1-33022287-1
  Warning  FailedMount  7m59s (x45 over 83m)  kubelet, aks-nodepool1-33022287-1  MountVolume.SetUp failed for volume "docker-push-secret" : secrets "binder-push-secret" not found
  Warning  FailedMount  118s (x36 over 81m)   kubelet, aks-nodepool1-33022287-1  Unable to mount volumes for pod "build-binder-2dexamples-2drequirements-55ab5c-a73ba1_default(0d3243e8-498f-11e9-8d40-fe8b4ef2ccd8)": timeout expired waiting for volumes to attach or mount for pod "default"/"build-binder-2dexamples-2drequirements-55ab5c-a73ba1". list of unmounted volumes=[docker-push-secret]. list of unattached volumes=[docker-socket docker-push-secret default-token-2fq72]

[ageorgou]

Trying this again from scratch, it seems to work (the containers are created in the expected namespace). I was following the same instructions, and the only difference I can see is in the description of the binder deployment. I now have

Annotations:            deployment.kubernetes.io/revision: 2

instead of

Annotations:            deployment.kubernetes.io/revision: 4

(it’s possible that I updated the deployment more times when I first tried this)

I guess I must have made a mistake somewhere along the way the first time, so this can be closed as far as I’m concerned!

[jtpio]

Just tried a new setup on OVH (new cluster and new BinderHub), but this time choosing 1.15 (the lowest they offer at the moment) as the Kubernetes version.

And it’s working out of the box without having to follow the steps mentioned in https://github.com/jupyterhub/binderhub/issues/810#issuecomment-651089134.

We should probably open a new issue to keep track of this. It looks like it is related to some rbac changes in the latest versions?