question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

K8s control plane logs conversion webhook errors

See original GitHub issue

Describe the bug After upgrading from Ambassador API gateway 1.13 to Emissary-ingress 2.2.2, the control plane logs have started showing the following errors continuously:

E0519 17:40:52.693825      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=Mapping: conversion webhook for getambassador.io/v2, Kind=Mapping failed: no kind "Mapping" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.538414      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=AuthService: conversion webhook for getambassador.io/v2, Kind=AuthService failed: no kind "AuthService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.332153      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=RateLimitService: conversion webhook for getambassador.io/v2, Kind=RateLimitService failed: no kind "RateLimitService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.180641      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=TCPMapping: conversion webhook for getambassador.io/v2, Kind=TCPMapping failed: no kind "TCPMapping" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.147282      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=TracingService: conversion webhook for getambassador.io/v2, Kind=TracingService failed: no kind "TracingService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:51.746911      12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=KubernetesEndpointResolver: conversion webhook for getambassador.io/v2, Kind=KubernetesEndpointResolver failed: no kind "KubernetesEndpointResolver" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...

Besides the error logs, everything seems to work correctly.

To Reproduce I don’t know how to reproduce this issue

Expected behavior I expect the control plane logs to not contain errors for getambassador.io/v1 objects.

Versions (please complete the following information):

  • Emissary-ingress: 2.2.2
  • Kubernetes AWS EKS v1.19.16-eks-25803e
  • Kubectl 1.23.5

Additional context We only have 1 tracingservice in our clusters.

First, it was a tracingservice version in v1 as an annotation

apiVersion: v1
kind: Service
metadata:
  name: zipkin
  namespace: monitoring
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: TracingService
      name: tracing
      service: zipkin.monitoring:9411
      driver: zipkin
...

We deleted the annotation and applied to the following v2 custom resource 1 year ago:

apiVersion: getambassador.io/v2
kind: TracingService
metadata:
  name: tracing
  namespace: monitoring
spec:
  ambassador_id:
    - internal
    - public
  driver: zipkin
  service: zipkin.monitoring:9411
  tag_headers:
    - :authority
    - :path

The number of error logs around 10700 log messages per 15 minutes, so its spams a lot.

Kubernetes CLI reports that there are stored versions of the CRD in v1 and v2 (see bottom lines):

$ kubectl get crd tracingservices.getambassador.io -o=yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.6.2"},"labels":{"app.kubernetes.io/instance":"emissary-apiext","app.kubernetes.io/managed-by":"kubectl_apply_-f_emissary-apiext.yaml","app.kubernetes.io/name":"emissary-apiext","app.kubernetes.io/part-of":"emissary-apiext","owner":"team_sre"},"name":"tracingservices.getambassador.io"},"spec":{"conversion":{"strategy":"Webhook","webhook":{"clientConfig":{"service":{"name":"emissary-apiext","namespace":"ambassador"}},"conversionReviewVersions":["v1beta1"]}},"group":"getambassador.io","names":{"categories":["ambassador-crds"],"kind":"TracingService","listKind":"TracingServiceList","plural":"tracingservices","singular":"tracingservice"},"preserveUnknownFields":false,"scope":"Namespaced","versions":[{"name":"v2","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TracingServiceSpec defines the desired state of TracingService","properties":{"config":{"properties":{"access_token_file":{"type":"string"},"collector_cluster":{"type":"string"},"collector_endpoint":{"type":"string"},"collector_endpoint_version":{"enum":["HTTP_JSON_V1","HTTP_JSON","HTTP_PROTO"],"type":"string"},"collector_hostname":{"type":"string"},"service_name":{"type":"string"},"shared_span_context":{"type":"boolean"},"trace_id_128bit":{"type":"boolean"}},"type":"object"},"driver":{"enum":["lightstep","zipkin","datadog"],"type":"string"},"sampling":{"properties":{"client":{"type":"integer"},"overall":{"type":"integer"},"random":{"type":"integer"}},"type":"object"},"service":{"type":"string"},"tag_headers":{"items":{"type":"string"},"type":"array"},"v3StatsName":{"type":"string"}},"required":["driver","service"],"type":"object","x-kubernetes-preserve-unknown-fields":true}},"type":"object"}},"served":true,"storage":true},{"name":"v3alpha1","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TracingServiceSpec defines the desired state of TracingService","properties":{"ambassador_id":{"description":"AmbassadorID declares which Ambassador instances should pay attention to this resource. If no value is provided, the default is: \n    ambassador_id:    - \"default\" \n TODO(lukeshu): In v3alpha2, consider renaming all of the `ambassador_id` (singular) fields to `ambassador_ids` (plural).","items":{"type":"string"},"type":"array"},"config":{"properties":{"access_token_file":{"type":"string"},"collector_cluster":{"type":"string"},"collector_endpoint":{"type":"string"},"collector_endpoint_version":{"enum":["HTTP_JSON_V1","HTTP_JSON","HTTP_PROTO"],"type":"string"},"collector_hostname":{"type":"string"},"service_name":{"type":"string"},"shared_span_context":{"type":"boolean"},"trace_id_128bit":{"type":"boolean"}},"type":"object"},"driver":{"enum":["lightstep","zipkin","datadog"],"type":"string"},"sampling":{"properties":{"client":{"type":"integer"},"overall":{"type":"integer"},"random":{"type":"integer"}},"type":"object"},"service":{"type":"string"},"stats_name":{"type":"string"},"tag_headers":{"items":{"type":"string"},"type":"array"}},"required":["driver","service"],"type":"object"}},"type":"object"}},"served":true,"storage":false},{"name":"v1","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","type":"object","x-kubernetes-preserve-unknown-fields":true}},"served":false,"storage":false}]}}
  creationTimestamp: "2019-09-12T09:15:35Z"
  generation: 12
  labels:
    app.kubernetes.io/instance: emissary-apiext
    app.kubernetes.io/managed-by: kubectl_apply_-f_emissary-apiext.yaml
    app.kubernetes.io/name: emissary-apiext
    app.kubernetes.io/part-of: emissary-apiext
    owner: team_sre
  name: tracingservices.getambassador.io
  resourceVersion: "2436618420"
  selfLink: /apis/apiextensions.k8s.io/v1/customresourcedefinitions/tracingservices.getambassador.io
  uid: e124e532-d53d-11e9-b8d3-0ab5b0c37404
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: <redacted>
        service:
          name: emissary-apiext
          namespace: ambassador
          path: /webhooks/crd-convert
          port: 443
      conversionReviewVersions:
      - v1beta1
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: TracingService
    listKind: TracingServiceList
    plural: tracingservices
    singular: tracingservice
  scope: Namespaced
  versions:
  - name: v2
    schema:
      openAPIV3Schema:
        description: TracingService is the Schema for the tracingservices API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TracingServiceSpec defines the desired state of TracingService
            properties:
              config:
                properties:
                  access_token_file:
                    type: string
                  collector_cluster:
                    type: string
                  collector_endpoint:
                    type: string
                  collector_endpoint_version:
                    enum:
                    - HTTP_JSON_V1
                    - HTTP_JSON
                    - HTTP_PROTO
                    type: string
                  collector_hostname:
                    type: string
                  service_name:
                    type: string
                  shared_span_context:
                    type: boolean
                  trace_id_128bit:
                    type: boolean
                type: object
              driver:
                enum:
                - lightstep
                - zipkin
                - datadog
                type: string
              sampling:
                properties:
                  client:
                    type: integer
                  overall:
                    type: integer
                  random:
                    type: integer
                type: object
              service:
                type: string
              tag_headers:
                items:
                  type: string
                type: array
              v3StatsName:
                type: string
            required:
            - driver
            - service
            type: object
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: true
  - name: v3alpha1
    schema:
      openAPIV3Schema:
        description: TracingService is the Schema for the tracingservices API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TracingServiceSpec defines the desired state of TracingService
            properties:
              ambassador_id:
                description: "AmbassadorID declares which Ambassador instances should
                  pay attention to this resource. If no value is provided, the default
                  is: \n    ambassador_id:    - \"default\" \n TODO(lukeshu): In v3alpha2,
                  consider renaming all of the `ambassador_id` (singular) fields to
                  `ambassador_ids` (plural)."
                items:
                  type: string
                type: array
              config:
                properties:
                  access_token_file:
                    type: string
                  collector_cluster:
                    type: string
                  collector_endpoint:
                    type: string
                  collector_endpoint_version:
                    enum:
                    - HTTP_JSON_V1
                    - HTTP_JSON
                    - HTTP_PROTO
                    type: string
                  collector_hostname:
                    type: string
                  service_name:
                    type: string
                  shared_span_context:
                    type: boolean
                  trace_id_128bit:
                    type: boolean
                type: object
              driver:
                enum:
                - lightstep
                - zipkin
                - datadog
                type: string
              sampling:
                properties:
                  client:
                    type: integer
                  overall:
                    type: integer
                  random:
                    type: integer
                type: object
              service:
                type: string
              stats_name:
                type: string
              tag_headers:
                items:
                  type: string
                type: array
            required:
            - driver
            - service
            type: object
        type: object
    served: true
    storage: false
  - name: v1
    schema:
      openAPIV3Schema:
        description: TracingService is the Schema for the tracingservices API
        type: object
        x-kubernetes-preserve-unknown-fields: true
    served: false
    storage: false
status:
  acceptedNames:
    categories:
    - ambassador-crds
    kind: TracingService
    listKind: TracingServiceList
    plural: tracingservices
    singular: tracingservice
  conditions:
  - lastTransitionTime: "2019-09-12T09:15:35Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1
  - v2

Kubectl only reports a single tracingservice and it is in v2:

$ kubectl get tracingservices.getambassador.io --all-namespaces -o=yaml

apiVersion: v1
items:
- apiVersion: getambassador.io/v2
  kind: TracingService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v2","kind":"TracingService","metadata":{"annotations":{},"name":"tracing","namespace":"monitoring"},"spec":{"ambassador_id":["public"],"driver":"zipkin","service":"zipkin.monitoring:9411","tag_headers":[":authority",":path"]}}
    creationTimestamp: "2021-03-03T22:19:56Z"
    generation: 2
    name: tracing
    namespace: monitoring
    resourceVersion: "791986493"
    selfLink: /apis/getambassador.io/v2/namespaces/monitoring/tracingservices/tracing
    uid: c69f7e0f-d621-4d52-b644-7bca965fba5f
  spec:
    ambassador_id:
    - public
    driver: zipkin
    service: zipkin.monitoring:9411
    tag_headers:
    - :authority
    - :path
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:4
  • Comments:5

github_iconTop GitHub Comments

2reactions
Alan01252commented, Jul 15, 2022

Same issue here, has anyone in this thread managed to resolve it?

Edit: https://github.com/emissary-ingress/emissary/issues/4224#issuecomment-1172535547

Seems removing the V1 from CRD does the trick.

0reactions
woz5999commented, Sep 30, 2022

Same issue. Got a nice fat logs bill as a reward for upgrading emissary.

Read more comments on GitHub >

github_iconTop Results From Across the Web

The Definitive Debugging Guide for the cert-manager ...
This guide helps you debug communication issues between the Kubernetes API server and the cert-manager webhook pod. The error messages ...
Read more >
Versions in CustomResourceDefinitions - Kubernetes
Versions can have different schemas, and conversion webhooks can convert custom resources between versions. Webhook conversions should follow the Kubernetes ...
Read more >
Add a conversion webhook to an operator to convert API ...
Step 2: Let's have a closer look at the changes ; import (. "k8s.io/apimachinery/pkg/runtime". ctrl "sigs.k8s.io/controller-runtime" ; logf "sigs.
Read more >
Ensuring compatibility of webhook certificates before ...
Starting from version 1.23, Kubernetes no longer supports server identity validation using the X.509 Common Name (CN) field in certificates.
Read more >
0 - Stack Overflow
kubectl get cluster issuers Output: Error from server: conversion webhook for cert-manager.io/v1alpha2, Kind=ClusterIssuer failed: an error ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found