K8s control plane logs conversion webhook errors

Describe the bug After upgrading from Ambassador API gateway 1.13 to Emissary-ingress 2.2.2, the control plane logs have started showing the following errors continuously:

E0519 17:40:52.693825      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=Mapping: conversion webhook for getambassador.io/v2, Kind=Mapping failed: no kind "Mapping" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.538414      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=AuthService: conversion webhook for getambassador.io/v2, Kind=AuthService failed: no kind "AuthService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.332153      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=RateLimitService: conversion webhook for getambassador.io/v2, Kind=RateLimitService failed: no kind "RateLimitService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.180641      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=TCPMapping: conversion webhook for getambassador.io/v2, Kind=TCPMapping failed: no kind "TCPMapping" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.147282      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=TracingService: conversion webhook for getambassador.io/v2, Kind=TracingService failed: no kind "TracingService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:51.746911      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=KubernetesEndpointResolver: conversion webhook for getambassador.io/v2, Kind=KubernetesEndpointResolver failed: no kind "KubernetesEndpointResolver" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...

Besides the error logs, everything seems to work correctly.

To Reproduce I don’t know how to reproduce this issue

Expected behavior I expect the control plane logs to not contain errors for getambassador.io/v1 objects.

Versions (please complete the following information):

• Emissary-ingress: 2.2.2
• Kubernetes AWS EKS v1.19.16-eks-25803e
• Kubectl 1.23.5

Additional context We only have 1 tracingservice in our clusters.

First, it was a tracingservice version in v1 as an annotation

apiVersion: v1
kind: Service
metadata:
  name: zipkin
  namespace: monitoring
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: TracingService
      name: tracing
      service: zipkin.monitoring:9411
      driver: zipkin
...

We deleted the annotation and applied to the following v2 custom resource 1 year ago:

apiVersion: getambassador.io/v2
kind: TracingService
metadata:
  name: tracing
  namespace: monitoring
spec:
  ambassador_id:
    - internal
    - public
  driver: zipkin
  service: zipkin.monitoring:9411
  tag_headers:
    - :authority
    - :path

The number of error logs around 10700 log messages per 15 minutes, so its spams a lot.

Kubernetes CLI reports that there are stored versions of the CRD in v1 and v2 (see bottom lines):

$ kubectl get crd tracingservices.getambassador.io -o=yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.6.2"},"labels":{"app.kubernetes.io/instance":"emissary-apiext","app.kubernetes.io/managed-by":"kubectl_apply_-f_emissary-apiext.yaml","app.kubernetes.io/name":"emissary-apiext","app.kubernetes.io/part-of":"emissary-apiext","owner":"team_sre"},"name":"tracingservices.getambassador.io"},"spec":{"conversion":{"strategy":"Webhook","webhook":{"clientConfig":{"service":{"name":"emissary-apiext","namespace":"ambassador"}},"conversionReviewVersions":["v1beta1"]}},"group":"getambassador.io","names":{"categories":["ambassador-crds"],"kind":"TracingService","listKind":"TracingServiceList","plural":"tracingservices","singular":"tracingservice"},"preserveUnknownFields":false,"scope":"Namespaced","versions":[{"name":"v2","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TracingServiceSpec defines the desired state of TracingService","properties":{"config":{"properties":{"access_token_file":{"type":"string"},"collector_cluster":{"type":"string"},"collector_endpoint":{"type":"string"},"collector_endpoint_version":{"enum":["HTTP_JSON_V1","HTTP_JSON","HTTP_PROTO"],"type":"string"},"collector_hostname":{"type":"string"},"service_name":{"type":"string"},"shared_span_context":{"type":"boolean"},"trace_id_128bit":{"type":"boolean"}},"type":"object"},"driver":{"enum":["lightstep","zipkin","datadog"],"type":"string"},"sampling":{"properties":{"client":{"type":"integer"},"overall":{"type":"integer"},"random":{"type":"integer"}},"type":"object"},"service":{"type":"string"},"tag_headers":{"items":{"type":"string"},"type":"array"},"v3StatsName":{"type":"string"}},"required":["driver","service"],"type":"object","x-kubernetes-preserve-unknown-fields":true}},"type":"object"}},"served":true,"storage":true},{"name":"v3alpha1","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TracingServiceSpec defines the desired state of TracingService","properties":{"ambassador_id":{"description":"AmbassadorID declares which Ambassador instances should pay attention to this resource. If no value is provided, the default is: \n    ambassador_id:    - \"default\" \n TODO(lukeshu): In v3alpha2, consider renaming all of the `ambassador_id` (singular) fields to `ambassador_ids` (plural).","items":{"type":"string"},"type":"array"},"config":{"properties":{"access_token_file":{"type":"string"},"collector_cluster":{"type":"string"},"collector_endpoint":{"type":"string"},"collector_endpoint_version":{"enum":["HTTP_JSON_V1","HTTP_JSON","HTTP_PROTO"],"type":"string"},"collector_hostname":{"type":"string"},"service_name":{"type":"string"},"shared_span_context":{"type":"boolean"},"trace_id_128bit":{"type":"boolean"}},"type":"object"},"driver":{"enum":["lightstep","zipkin","datadog"],"type":"string"},"sampling":{"properties":{"client":{"type":"integer"},"overall":{"type":"integer"},"random":{"type":"integer"}},"type":"object"},"service":{"type":"string"},"stats_name":{"type":"string"},"tag_headers":{"items":{"type":"string"},"type":"array"}},"required":["driver","service"],"type":"object"}},"type":"object"}},"served":true,"storage":false},{"name":"v1","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","type":"object","x-kubernetes-preserve-unknown-fields":true}},"served":false,"storage":false}]}}
  creationTimestamp: "2019-09-12T09:15:35Z"
  generation: 12
  labels:
    app.kubernetes.io/instance: emissary-apiext
    app.kubernetes.io/managed-by: kubectl_apply_-f_emissary-apiext.yaml
    app.kubernetes.io/name: emissary-apiext
    app.kubernetes.io/part-of: emissary-apiext
    owner: team_sre
  name: tracingservices.getambassador.io
  resourceVersion: "2436618420"
  selfLink: /apis/apiextensions.k8s.io/v1/customresourcedefinitions/tracingservices.getambassador.io
  uid: e124e532-d53d-11e9-b8d3-0ab5b0c37404
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: <redacted>
        service:
          name: emissary-apiext
          namespace: ambassador
          path: /webhooks/crd-convert
          port: 443
      conversionReviewVersions:
      - v1beta1
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: TracingService
    listKind: TracingServiceList
    plural: tracingservices
    singular: tracingservice
  scope: Namespaced
  versions:
  - name: v2
    schema:
      openAPIV3Schema:
        description: TracingService is the Schema for the tracingservices API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TracingServiceSpec defines the desired state of TracingService
            properties:
              config:
                properties:
                  access_token_file:
                    type: string
                  collector_cluster:
                    type: string
                  collector_endpoint:
                    type: string
                  collector_endpoint_version:
                    enum:
                    - HTTP_JSON_V1
                    - HTTP_JSON
                    - HTTP_PROTO
                    type: string
                  collector_hostname:
                    type: string
                  service_name:
                    type: string
                  shared_span_context:
                    type: boolean
                  trace_id_128bit:
                    type: boolean
                type: object
              driver:
                enum:
                - lightstep
                - zipkin
                - datadog
                type: string
              sampling:
                properties:
                  client:
                    type: integer
                  overall:
                    type: integer
                  random:
                    type: integer
                type: object
              service:
                type: string
              tag_headers:
                items:
                  type: string
                type: array
              v3StatsName:
                type: string
            required:
            - driver
            - service
            type: object
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: true
  - name: v3alpha1
    schema:
      openAPIV3Schema:
        description: TracingService is the Schema for the tracingservices API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TracingServiceSpec defines the desired state of TracingService
            properties:
              ambassador_id:
                description: "AmbassadorID declares which Ambassador instances should
                  pay attention to this resource. If no value is provided, the default
                  is: \n    ambassador_id:    - \"default\" \n TODO(lukeshu): In v3alpha2,
                  consider renaming all of the `ambassador_id` (singular) fields to
                  `ambassador_ids` (plural)."
                items:
                  type: string
                type: array
              config:
                properties:
                  access_token_file:
                    type: string
                  collector_cluster:
                    type: string
                  collector_endpoint:
                    type: string
                  collector_endpoint_version:
                    enum:
                    - HTTP_JSON_V1
                    - HTTP_JSON
                    - HTTP_PROTO
                    type: string
                  collector_hostname:
                    type: string
                  service_name:
                    type: string
                  shared_span_context:
                    type: boolean
                  trace_id_128bit:
                    type: boolean
                type: object
              driver:
                enum:
                - lightstep
                - zipkin
                - datadog
                type: string
              sampling:
                properties:
                  client:
                    type: integer
                  overall:
                    type: integer
                  random:
                    type: integer
                type: object
              service:
                type: string
              stats_name:
                type: string
              tag_headers:
                items:
                  type: string
                type: array
            required:
            - driver
            - service
            type: object
        type: object
    served: true
    storage: false
  - name: v1
    schema:
      openAPIV3Schema:
        description: TracingService is the Schema for the tracingservices API
        type: object
        x-kubernetes-preserve-unknown-fields: true
    served: false
    storage: false
status:
  acceptedNames:
    categories:
    - ambassador-crds
    kind: TracingService
    listKind: TracingServiceList
    plural: tracingservices
    singular: tracingservice
  conditions:
  - lastTransitionTime: "2019-09-12T09:15:35Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1
  - v2

Kubectl only reports a single tracingservice and it is in v2:

$ kubectl get tracingservices.getambassador.io --all-namespaces -o=yaml

apiVersion: v1
items:
- apiVersion: getambassador.io/v2
  kind: TracingService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v2","kind":"TracingService","metadata":{"annotations":{},"name":"tracing","namespace":"monitoring"},"spec":{"ambassador_id":["public"],"driver":"zipkin","service":"zipkin.monitoring:9411","tag_headers":[":authority",":path"]}}
    creationTimestamp: "2021-03-03T22:19:56Z"
    generation: 2
    name: tracing
    namespace: monitoring
    resourceVersion: "791986493"
    selfLink: /apis/getambassador.io/v2/namespaces/monitoring/tracingservices/tracing
    uid: c69f7e0f-d621-4d52-b644-7bca965fba5f
  spec:
    ambassador_id:
    - public
    driver: zipkin
    service: zipkin.monitoring:9411
    tag_headers:
    - :authority
    - :path
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[Alan01252]

Same issue here, has anyone in this thread managed to resolve it?

Edit: https://github.com/emissary-ingress/emissary/issues/4224#issuecomment-1172535547

Seems removing the V1 from CRD does the trick.

[woz5999]

Same issue. Got a nice fat logs bill as a reward for upgrading emissary.