Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Compatibility with Kubernetes 1.18 on OVH

See original GitHub issue

Bug description

Hey folks,

Creating a new issue from several comments in https://github.com/jupyterhub/binderhub/issues/810#issuecomment-510020306 for better visibility.

This is a summary of the issues encountered with a fresh BinderHub install on a new Kubernetes 1.18 cluster on OVH.

Expected behaviour

BinderHub should be able to build binders out of the box after following the instructions from the Zero to BinderHub guide.

Actual behaviour

From this comment (July 2019): https://github.com/jupyterhub/binderhub/issues/810#issuecomment-510020306

Note: it’s possible that the Kubernetes version used at that time back in July 2019 was not 1.18, but maybe 1.17 (can’t remember).

The binder pod uses the wrong Kubernetes namespace. From the pod logs:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:binderhub:binderhub\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

The release is deployed in the binderhub namespace, but binder wants to list the pods in the default namespace, even though BUILD_NAMESPACE is correctly set to binderhub (the namespace where the chart is released):

$ kubectl exec -it binder-d998c657c-zmdf8 -- env | grep BUILD_NAMESPACE
BUILD_NAMESPACE=binderhub

root@binder-d998c657c-zmdf8:/# tr '\0' '\n' < /proc/1/environ | grep BUILD_NAMESPACE
BUILD_NAMESPACE=binderhub

The logs for the binder pod:

[E 190710 10:48:12 app:638] Failed to cleanup build pods
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/binderhub/app.py", line 630, in watch_build_pods
        lambda: Build.cleanup_builds(
      File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.6/site-packages/binderhub/app.py", line 633, in <lambda>
        self.build_max_age,
      File "/usr/local/lib/python3.6/site-packages/binderhub/build.py", line 91, in cleanup_builds
        label_selector='component=binderhub-build',
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod
        (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
        _return_http_data_only, collection_formats, _preload_content, _request_timeout)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 342, in request
        headers=headers)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in GET
        query_params=query_params)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (403)
    Reason: Forbidden
    HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 10 Jul 2019 10:48:12 GMT', 'Content-Length': '286'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:binderhub:binderhub\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Looking at the code, it looks like build_namespace should be correctly passed:

https://github.com/jupyterhub/binderhub/blob/01b1c59b9e7dc81250c1ed579c492ec2fd6baaf6/binderhub/app.py#L630-L634

The issue mentioned above seems to correspond to the step that comes after the binderhub serviceaccount has the correct rights on the default namespace.

From this comment (June 2020): https://github.com/jupyterhub/binderhub/issues/810#issuecomment-651089134

Trying a fresh BinderHub deployment on a new Kubernetes cluster.

Setting:

config:
  BinderHub:
    auth_enabled: false

helps get past the following error in /binderhub_config.py:

Loading /etc/binderhub/config/values.yaml
Loading /etc/binderhub/secret/values.yaml
[BinderHub] ERROR | Exception while loading config file /binderhub_config.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/traitlets/config/application.py", line 563, in _load_config_files
    config = loader.load_config()
  File "/usr/local/lib/python3.7/site-packages/traitlets/config/loader.py", line 457, in load_config
    self._read_file_as_dict()
  File "/usr/local/lib/python3.7/site-packages/traitlets/config/loader.py", line 489, in _read_file_as_dict
    py3compat.execfile(conf_filename, namespace)
  File "/usr/local/lib/python3.7/site-packages/ipython_genutils/py3compat.py", line 198, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
  File "/binderhub_config.py", line 87, in <module>
    hub_url = urlparse(c.BinderHub.hub_url)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 367, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 123, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 107, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 107, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'LazyConfigValue' object has no attribute 'decode'

Setting:

extraConfig:
  01-custom: |
      c.BinderHub.build_namespace = "binderhub"

helps prevent Binder trying to list pods in the default namespace.

However permissions still seem to be off even though the namespace looks good this time:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:binderhub:binderhub\" cannot list resource \"pods\" in API group \"\" in the namespace \"binderhub\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

The binderhub Role and RoleBinding also look good and correspond to the defaults from the chart:

https://github.com/jupyterhub/binderhub/blob/58a0b72021d17264519438f6e06f452021617a35/helm-chart/binderhub/templates/rbac.yaml#L2-L34

Creating a new Role and RoleBinding with all the permissions on the binderhub namespace for the binderhub ServiceAccount doesn’t change anything.

How to reproduce

Create a new Kubernetes cluster on OVH, choosing 1.18 as the k8s version
Follow the Zero to BinderHub guide
Go to the BinderHub landing page and try to build a new repo

It is unclear whether this is an issue with Kubernetes 1.18, or with the cloud vendor. GKE doesn’t seem to offer 1.18 yet and can’t be used as a comparison.

Could it be related to a recent change to rbac in recent Kubernetes versions?