GCE - K8s 1.8 - pods is forbidden - Cannot list pods - Unknown user "system:serviceaccount:default:default"

I used the mongo-k8s-sidecar for somewhilte and everythign worked fine.

today i created a new cluster with 1.8.7-gke.1 and deployed mongo db with the sicdecar. As a result the pods don’t get connected (see below) when i recreate the cluster with k8s 1.7.12-gke.1 everything works fine.

From sidecar log when faliing: message: ‘pods is forbidden: User “system:serviceaccount:default:default” cannot list pods at the cluster scope: Unknown user “system:serviceaccount:default:default”’,

mongo-sidecar | Feb 28, 2018, 11:04:19 AM | status: ‘Failure’, mongo-sidecar | Feb 28, 2018, 11:04:19 AM | metadata: {}, mongo-sidecar | Feb 28, 2018, 11:04:19 AM | apiVersion: ‘v1’, mongo-sidecar | Feb 28, 2018, 11:04:19 AM | { kind: ‘Status’, mongo-sidecar | Feb 28, 2018, 11:04:19 AM | message: mongo-sidecar | Feb 28, 2018, 11:04:19 AM | Error in workloop { [Error: [object Object]] mongo-sidecar | Feb 28, 2018, 11:04:14 AM | statusCode: 403 } mongo-sidecar | Feb 28, 2018, 11:04:14 AM | code: 403 }, mongo-sidecar | Feb 28, 2018, 11:04:14 AM | details: { kind: ‘pods’ }, mongo-sidecar | Feb 28, 2018, 11:04:14 AM | reason: ‘Forbidden’,

any hints what has changed in gce permission system? Something i need to configure?

configuration

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
---
apiVersion: v1
kind: Service
metadata:
  name: mongo-lp
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo-lp
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo-lp-statefulset
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo-lp
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:3.4.9
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage-lp
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo-lp,environment=test"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage-lp
      annotations:
        volume.beta.kubernetes.io/storage-class: "fast"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 32Gi

[westonpace]

One simple fix is to create the following role binding:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: default-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default

This will grant the default service account view permissions. A more secure approach would probably be to create a new service account, grant it the view permissions, and then assign that service account to the sidecar.

[mauri870]

@grohan2002 By now you probably have figured out, but for the sake of completeness then yes, you need to restart the pods for this to take effect.