Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Scaling behavior permission in namespace issues

See original GitHub issue

I’ve got a cluster up and running with dask_kubernetes installed. I am using this as a guide:

When I go to initialize my scaling behavior I get the following error:

from dask_kubernetes import KubeCluster
cluster = KubeCluster.from_yaml('worker-spec.yml')
cluster.scale_up(3)

....
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7d4dba0b-e4e2-40c8-bedc-33e93a3801a1', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Tue, 20 Mar 2018 00:45:44 GMT', 'Content-Length': '307'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:default:default\" cannot list pods in the namespace \"default\": Unknown user \"system:serviceaccount:default:default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

My worker-spec.yml looks like this:

kind: Pod
metadata:
spec:
  restartPolicy: Never
  containers:
  - image: daskdev/dask:latest
    name: worker
    args: [--nthreads, '2', --no-bokeh, --memory-limit, 6GB, --death-timeout, '60']
    env:
      - name: EXTRA_PIP_PACKAGES
        value: fastparquet git+https://github.com/dask/distributed

This is the same behavior as https://github.com/pangeo-data/pangeo/issues/167
I have copied dask-kubernetes-serviceaccount.yml file and run kubectl create -f dask-kubernetes-serviceaccount.yml.

Is there anything obvious I am missing?

Issue Analytics

State:
Created 6 years ago
Comments:15 (8 by maintainers)

Top GitHub Comments

1reaction

mrocklincommented, Mar 21, 2018

@jgerardsimcock my guess is that the Jupyter pod from which you are trying to use dask-kubernetes still does not have sufficient permissions to launch pods. My guess is that you probably still need to futz with roles to make things work well. Unfortunately I don’t have much experience here, hopefully others present can help more.

1reaction

tjcronecommented, Mar 21, 2018

Sorry, but I have not seen this behavior before. However I do not run dask-kubernetes without jupyterhub, so hopefully someone else can chime in with suggestions.

Top Results From Across the Web

Namespace Permissions (Kubernetes RBAC-based) - 华为云

You can regulate users' or user groups' access to Kubernetes resources in a single namespace based on their Kubernetes RBAC roles.

Kubernetes Role-Based Access Control (RBAC) - StrongDM

This role can create roles and role bindings within a particular namespace. It does not permit write access to the namespace itself.

Limiting access to Kubernetes resources with RBAC - Learnk8s

Learn how to recreate the Kubernetes RBAC authorization model from scratch and practice the relationships between Roles, ServiceAccounts, RoleBindings, etc.

Troubleshoot service auto scaling issues in Amazon ECS

The intention of Application Auto Scaling is to prioritize the availability. Therefore, the behavior of these policies differs depending on ...

Namespace Permission Set Licenses in Packages

If you install a managed package that has an associated namespace permission set license, you can use it to entitle users access to...