linkerd serves requests for deleted Kube Service for 15+ minutes

Issue Type:

• Bug report

What happened: I deleted a Kubernetes Service resource that was backing a Kubernetes Ingress resource. Requests to the Kubernetes ingress resource continued to work fine for approximately 15-20 minutes after the deletion.

What you expected to happen: I expected linkerd to notice the Service resource had been deleted and be unable to route the Ingress traffic.

How to reproduce it (as minimally and precisely as possible):

1. Deploy linkerd on Kubernetes (see below for configs)
2. Deploy nginx backends, service, and ingress (see below for configs)
3. kubectl delete svc nginx

Anything else we need to know?: I suspect this is the same issue, or related to, #1730. I wanted to raise a new issue to avoid polluting that one with my config dumps.

Environment:

• linkerd/namerd version, config files: linkerd v1.3.5.
• Platform, version: Kubernetes 1.9.2
• Cloud provider or hardware configuration: GCE, running bespoke Kubernetes deployment (not GKE)

Configurations linkerd:

    admin:
      ip: 0.0.0.0
      port: 9990

    namers:
    - kind: io.l5d.k8s
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.http
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: kube-system
        port: http-incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.h2
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: kube-system
        port: h2-incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.grpc
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: kube-system
        port: grpc-incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.rewrite
      prefix: /portNsSvcToK8s
      pattern: "/{port}/{ns}/{svc}"
      name: "/k8s/{ns}/{port}/{svc}"

    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.05
    - kind: io.zipkin.http
      host: zipkin:9411
      initialSampleRate: 0.05

    usage:
      enabled: false

    routers:
    - label: http-outgoing
      originator: true
      protocol: http
      servers:
      - port: 4140
        ip: 0.0.0.0
        maxConcurrentRequests: 5000
      dtab: |
        /ph  => /$/io.buoyant.rinet ;                     # /ph/80/google.com -> /$/io.buoyant.rinet/80/google.com
        /svc => /ph/80 ;                                  # /svc/google.com -> /ph/80/google.com
        /svc => /$/io.buoyant.porthostPfx/ph ;            # /svc/google.com:80 -> /ph/80/google.com
        /k8s => /#/io.l5d.k8s.http ;                      # /k8s/default/http/foo -> /#/io.l5d.k8s.http/default/http/foo
        /portNsSvc => /#/portNsSvcToK8s ;                 # /portNsSvc/http/default/foo -> /k8s/default/http/foo
        /host => /portNsSvc/http/default ;                # /host/foo -> /portNsSvc/http/default/foo
        /host => /portNsSvc/http ;                        # /host/default/foo -> /portNsSvc/http/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ; # /svc/foo.default -> /host/default/foo
      client:
        kind: io.l5d.static
        configs:
        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            commonName: "{service}"

    - label: http-incoming
      protocol: http
      servers:
      - port: 4141
        ip: 0.0.0.0
        maxConcurrentRequests: 5000
      identifier:
        - kind: io.l5d.ingress
        - kind: io.l5d.header.token
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      dtab: |
        /svc => /#/io.l5d.k8s ;                           # /svc/default/http/foo -> /#/io.l5d.k8s/default/http/foo
        /k8s => /#/io.l5d.k8s ;                           # /k8s/default/http/foo -> /#/io.l5d.k8s/default/http/foo
        /portNsSvc => /#/portNsSvcToK8s ;                 # /portNsSvc/http/default/foo -> /k8s/default/http/foo
        /host => /portNsSvc/http/default ;                # /host/foo -> /portNsSvc/http/default/foo
        /host => /portNsSvc/http ;                        # /host/default/foo -> /portNsSvc/http/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ; # /svc/foo.default -> /host/default/foo

    - label: h2-outgoing
      originator: true
      protocol: h2
      servers:
      - port: 4240
        ip: 0.0.0.0
        maxConcurrentRequests: 5000
      dtab: |
        /ph  => /$/io.buoyant.rinet ;                       # /ph/80/google.com -> /$/io.buoyant.rinet/80/google.com
        /svc => /ph/80 ;                                    # /svc/google.com -> /ph/80/google.com
        /svc => /$/io.buoyant.porthostPfx/ph ;              # /svc/google.com:80 -> /ph/80/google.com
        /k8s => /#/io.l5d.k8s.h2 ;                          # /k8s/default/h2/foo -> /#/io.l5d.k8s.h2/default/h2/foo
        /portNsSvc => /#/portNsSvcToK8s ;                   # /portNsSvc/h2/default/foo -> /k8s/default/h2/foo
        /host => /portNsSvc/h2/default ;                    # /host/foo -> /portNsSvc/h2/default/foo
        /host => /portNsSvc/h2 ;                            # /host/default/foo -> /portNsSvc/h2/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ;   # /svc/foo.default -> /host/default/foo
      client:
        kind: io.l5d.static
        configs:
        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            commonName: "{service}"

    - label: h2-incoming
      protocol: h2
      servers:
      - port: 4241
        ip: 0.0.0.0
        maxConcurrentRequests: 5000
      identifier:
        - kind: io.l5d.ingress
        - kind: io.l5d.header.token
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      dtab: |
        /svc => /#/io.l5d.k8s ;                             # /svc/default/h2/foo -> /#/io.l5d.k8s/default/h2/foo
        /k8s => /#/io.l5d.k8s ;                             # /k8s/default/h2/foo -> /#/io.l5d.k8s/default/h2/foo
        /portNsSvc => /#/portNsSvcToK8s ;                   # /portNsSvc/h2/default/foo -> /k8s/default/h2/foo
        /host => /portNsSvc/h2/default ;                    # /host/foo -> /portNsSvc/h2/default/foo
        /host => /portNsSvc/h2 ;                            # /host/default/foo -> /portNsSvc/h2/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ;   # /svc/foo.default -> /host/default/foo

    - label: grpc-outgoing
      originator: true
      protocol: h2
      servers:
      - port: 4340
        ip: 0.0.0.0
        maxConcurrentRequests: 5000
      identifier:
        kind: io.l5d.header.path
        segments: 1
      dtab: |
        /hp  => /$/inet ;                                # /hp/linkerd.io/8888 -> /$/inet/linkerd.io/8888
        /svc => /$/io.buoyant.hostportPfx/hp ;           # /svc/linkerd.io:8888 -> /hp/linkerd.io/8888
        /srv => /#/io.l5d.k8s.grpc/default/grpc;         # /srv/service/package -> /#/io.l5d.k8s.grpc/default/grpc/service/package
        /svc => /$/io.buoyant.http.domainToPathPfx/srv ; # /svc/package.service -> /srv/service/package
      client:
        kind: io.l5d.static
        configs:
        - prefix: "/$/inet/{service}"
          tls:
            commonName: "{service}"

    - label: gprc-incoming
      protocol: h2
      servers:
      - port: 4341
        ip: 0.0.0.0
        maxConcurrentRequests: 5000
      identifier:
        kind: io.l5d.header.path
        segments: 1
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      dtab: |
        /srv => /#/io.l5d.k8s/default/grpc ;             # /srv/service/package -> /#/io.l5d.k8s/default/grpc/service/package
        /svc => /$/io.buoyant.http.domainToPathPfx/srv ; # /svc/package.service -> /srv/service/package

    - protocol: http
      label: http-ingress
      originator: true
      servers:
        - port: 80
          ip: 0.0.0.0
          maxConcurrentRequests: 5000
      identifier:
        kind: io.l5d.ingress
      dtab: /svc => /#/io.l5d.k8s.http ;                 # /svc/default/http/foo -> /#/io.l5d.k8s/default/http/foo

    - protocol: h2
      originator: true
      label: h2-ingress
      servers:
        - port: 81
          ip: 0.0.0.0
          maxConcurrentRequests: 5000
      identifier:
        kind: io.l5d.ingress
      dtab: /svc => /#/io.l5d.k8s.h2 ;                   # /svc/default/h2/foo -> /#/io.l5d.k8s/default/h2/foo

Kubernetes ingress, service, etc:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
data:
  index.html: |-
    CRITICAL BUSINESS LOGIC!
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  labels:
    component: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      component: nginx
  template:
    metadata:
      name: nginx
      labels:
        component: nginx
    spec:
      volumes:
      - name: nginx
        configMap:
          name: "nginx"
      containers:
      - name: nginx
        image: nginx:alpine
        ports:           
        - name: nginx
          containerPort: 80
        volumeMounts:
        - name: "nginx"
          mountPath: "/usr/share/nginx/html"
          readOnly: true
        resources:
          limits:
            cpu: 500m
            memory: 500Mi
          requests:
            cpu: 500m
            memory: 500Mi
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: nginx
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      component: nginx
---
# This is the service I deleted.
apiVersion: v1
kind: Service
metadata:
  labels:
    name: nginx
  name: nginx
spec:
  selector:
    component: nginx
  ports:
  - name: nginx
    protocol: TCP
    port: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
  labels:
    component: nginx
  name: nginx
spec:
  rules:
  - host: nginx.ingress.REDACTED
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

[negz]

@deebo91 A little more (unfortunately anecdotal and unreproducible) context - we’ve also seen the reverse of this issue at least once. i.e. We created a Service resource and linkerd did not notice.

Specifically we:

1. Setup a Kubernetes cluster and deployed a dummy nginx Service and Ingress resource.
2. Confirmed it successfully served requests via the Ingress.
3. Deleted the Service resource.
4. Confirmed that linkerd could not route requests for the now Service-less Ingress.
5. Reinstated the Service resource ~30 mins later.
6. Observed that linkerd continued to not route requests for the Ingress.

It’s possible given #1810’s bug that:

1. linkerd sees a DELETE watch event for the Service. DELETE events do not increment the resourceVersion because they return “the state of the object immediately before deletion”.
2. linkerd sees an ADDED watch event for the same Service. The ADDED resource has a resourceVersion lower than the previously DELETED Service, and thus get swallowed?

Though in that case I’d have expected linkerd to never have noticed the deletion in the first place? 🤔

From the Kubernetes API WatchEvent docs:

If Type is Deleted: the state of the object immediately before deletion

[siggy]

Fixed in #1810. Please re-open if this issue resurfaces.