K8s control plane logs conversion webhook errors
See original GitHub issueDescribe the bug After upgrading from Ambassador API gateway 1.13 to Emissary-ingress 2.2.2, the control plane logs have started showing the following errors continuously:
E0519 17:40:52.693825 12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=Mapping: conversion webhook for getambassador.io/v2, Kind=Mapping failed: no kind "Mapping" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.538414 12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=AuthService: conversion webhook for getambassador.io/v2, Kind=AuthService failed: no kind "AuthService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.332153 12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=RateLimitService: conversion webhook for getambassador.io/v2, Kind=RateLimitService failed: no kind "RateLimitService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.180641 12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=TCPMapping: conversion webhook for getambassador.io/v2, Kind=TCPMapping failed: no kind "TCPMapping" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:52.147282 12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=TracingService: conversion webhook for getambassador.io/v2, Kind=TracingService failed: no kind "TracingService" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
E0519 17:40:51.746911 12 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list getambassador.io/v1, Kind=KubernetesEndpointResolver: conversion webhook for getambassador.io/v2, Kind=KubernetesEndpointResolver failed: no kind "KubernetesEndpointResolver" is registered for version "getambassador.io/v1" in scheme "/go/cmd/apiext/main.go:50"; reinitializing...
Besides the error logs, everything seems to work correctly.
To Reproduce I don’t know how to reproduce this issue
Expected behavior I expect the control plane logs to not contain errors for getambassador.io/v1 objects.
Versions (please complete the following information):
- Emissary-ingress: 2.2.2
- Kubernetes AWS EKS v1.19.16-eks-25803e
- Kubectl 1.23.5
Additional context We only have 1 tracingservice in our clusters.
First, it was a tracingservice version in v1 as an annotation
apiVersion: v1
kind: Service
metadata:
name: zipkin
namespace: monitoring
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: TracingService
name: tracing
service: zipkin.monitoring:9411
driver: zipkin
...
We deleted the annotation and applied to the following v2 custom resource 1 year ago:
apiVersion: getambassador.io/v2
kind: TracingService
metadata:
name: tracing
namespace: monitoring
spec:
ambassador_id:
- internal
- public
driver: zipkin
service: zipkin.monitoring:9411
tag_headers:
- :authority
- :path
The number of error logs around 10700 log messages per 15 minutes, so its spams a lot.
Kubernetes CLI reports that there are stored versions of the CRD in v1 and v2 (see bottom lines):
$ kubectl get crd tracingservices.getambassador.io -o=yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.6.2
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.6.2"},"labels":{"app.kubernetes.io/instance":"emissary-apiext","app.kubernetes.io/managed-by":"kubectl_apply_-f_emissary-apiext.yaml","app.kubernetes.io/name":"emissary-apiext","app.kubernetes.io/part-of":"emissary-apiext","owner":"team_sre"},"name":"tracingservices.getambassador.io"},"spec":{"conversion":{"strategy":"Webhook","webhook":{"clientConfig":{"service":{"name":"emissary-apiext","namespace":"ambassador"}},"conversionReviewVersions":["v1beta1"]}},"group":"getambassador.io","names":{"categories":["ambassador-crds"],"kind":"TracingService","listKind":"TracingServiceList","plural":"tracingservices","singular":"tracingservice"},"preserveUnknownFields":false,"scope":"Namespaced","versions":[{"name":"v2","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TracingServiceSpec defines the desired state of TracingService","properties":{"config":{"properties":{"access_token_file":{"type":"string"},"collector_cluster":{"type":"string"},"collector_endpoint":{"type":"string"},"collector_endpoint_version":{"enum":["HTTP_JSON_V1","HTTP_JSON","HTTP_PROTO"],"type":"string"},"collector_hostname":{"type":"string"},"service_name":{"type":"string"},"shared_span_context":{"type":"boolean"},"trace_id_128bit":{"type":"boolean"}},"type":"object"},"driver":{"enum":["lightstep","zipkin","datadog"],"type":"string"},"sampling":{"properties":{"client":{"type":"integer"},"overall":{"type":"integer"},"random":{"type":"integer"}},"type":"object"},"service":{"type":"string"},"tag_headers":{"items":{"type":"string"},"type":"array"},"v3StatsName":{"type":"string"}},"required":["driver","service"],"type":"object","x-kubernetes-preserve-unknown-fields":true}},"type":"object"}},"served":true,"storage":true},{"name":"v3alpha1","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TracingServiceSpec defines the desired state of TracingService","properties":{"ambassador_id":{"description":"AmbassadorID declares which Ambassador instances should pay attention to this resource. If no value is provided, the default is: \n ambassador_id: - \"default\" \n TODO(lukeshu): In v3alpha2, consider renaming all of the `ambassador_id` (singular) fields to `ambassador_ids` (plural).","items":{"type":"string"},"type":"array"},"config":{"properties":{"access_token_file":{"type":"string"},"collector_cluster":{"type":"string"},"collector_endpoint":{"type":"string"},"collector_endpoint_version":{"enum":["HTTP_JSON_V1","HTTP_JSON","HTTP_PROTO"],"type":"string"},"collector_hostname":{"type":"string"},"service_name":{"type":"string"},"shared_span_context":{"type":"boolean"},"trace_id_128bit":{"type":"boolean"}},"type":"object"},"driver":{"enum":["lightstep","zipkin","datadog"],"type":"string"},"sampling":{"properties":{"client":{"type":"integer"},"overall":{"type":"integer"},"random":{"type":"integer"}},"type":"object"},"service":{"type":"string"},"stats_name":{"type":"string"},"tag_headers":{"items":{"type":"string"},"type":"array"}},"required":["driver","service"],"type":"object"}},"type":"object"}},"served":true,"storage":false},{"name":"v1","schema":{"openAPIV3Schema":{"description":"TracingService is the Schema for the tracingservices API","type":"object","x-kubernetes-preserve-unknown-fields":true}},"served":false,"storage":false}]}}
creationTimestamp: "2019-09-12T09:15:35Z"
generation: 12
labels:
app.kubernetes.io/instance: emissary-apiext
app.kubernetes.io/managed-by: kubectl_apply_-f_emissary-apiext.yaml
app.kubernetes.io/name: emissary-apiext
app.kubernetes.io/part-of: emissary-apiext
owner: team_sre
name: tracingservices.getambassador.io
resourceVersion: "2436618420"
selfLink: /apis/apiextensions.k8s.io/v1/customresourcedefinitions/tracingservices.getambassador.io
uid: e124e532-d53d-11e9-b8d3-0ab5b0c37404
spec:
conversion:
strategy: Webhook
webhook:
clientConfig:
caBundle: <redacted>
service:
name: emissary-apiext
namespace: ambassador
path: /webhooks/crd-convert
port: 443
conversionReviewVersions:
- v1beta1
group: getambassador.io
names:
categories:
- ambassador-crds
kind: TracingService
listKind: TracingServiceList
plural: tracingservices
singular: tracingservice
scope: Namespaced
versions:
- name: v2
schema:
openAPIV3Schema:
description: TracingService is the Schema for the tracingservices API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: TracingServiceSpec defines the desired state of TracingService
properties:
config:
properties:
access_token_file:
type: string
collector_cluster:
type: string
collector_endpoint:
type: string
collector_endpoint_version:
enum:
- HTTP_JSON_V1
- HTTP_JSON
- HTTP_PROTO
type: string
collector_hostname:
type: string
service_name:
type: string
shared_span_context:
type: boolean
trace_id_128bit:
type: boolean
type: object
driver:
enum:
- lightstep
- zipkin
- datadog
type: string
sampling:
properties:
client:
type: integer
overall:
type: integer
random:
type: integer
type: object
service:
type: string
tag_headers:
items:
type: string
type: array
v3StatsName:
type: string
required:
- driver
- service
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
served: true
storage: true
- name: v3alpha1
schema:
openAPIV3Schema:
description: TracingService is the Schema for the tracingservices API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: TracingServiceSpec defines the desired state of TracingService
properties:
ambassador_id:
description: "AmbassadorID declares which Ambassador instances should
pay attention to this resource. If no value is provided, the default
is: \n ambassador_id: - \"default\" \n TODO(lukeshu): In v3alpha2,
consider renaming all of the `ambassador_id` (singular) fields to
`ambassador_ids` (plural)."
items:
type: string
type: array
config:
properties:
access_token_file:
type: string
collector_cluster:
type: string
collector_endpoint:
type: string
collector_endpoint_version:
enum:
- HTTP_JSON_V1
- HTTP_JSON
- HTTP_PROTO
type: string
collector_hostname:
type: string
service_name:
type: string
shared_span_context:
type: boolean
trace_id_128bit:
type: boolean
type: object
driver:
enum:
- lightstep
- zipkin
- datadog
type: string
sampling:
properties:
client:
type: integer
overall:
type: integer
random:
type: integer
type: object
service:
type: string
stats_name:
type: string
tag_headers:
items:
type: string
type: array
required:
- driver
- service
type: object
type: object
served: true
storage: false
- name: v1
schema:
openAPIV3Schema:
description: TracingService is the Schema for the tracingservices API
type: object
x-kubernetes-preserve-unknown-fields: true
served: false
storage: false
status:
acceptedNames:
categories:
- ambassador-crds
kind: TracingService
listKind: TracingServiceList
plural: tracingservices
singular: tracingservice
conditions:
- lastTransitionTime: "2019-09-12T09:15:35Z"
message: no conflicts found
reason: NoConflicts
status: "True"
type: NamesAccepted
- lastTransitionTime: null
message: the initial names have been accepted
reason: InitialNamesAccepted
status: "True"
type: Established
storedVersions:
- v1
- v2
Kubectl only reports a single tracingservice and it is in v2:
$ kubectl get tracingservices.getambassador.io --all-namespaces -o=yaml
apiVersion: v1
items:
- apiVersion: getambassador.io/v2
kind: TracingService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"getambassador.io/v2","kind":"TracingService","metadata":{"annotations":{},"name":"tracing","namespace":"monitoring"},"spec":{"ambassador_id":["public"],"driver":"zipkin","service":"zipkin.monitoring:9411","tag_headers":[":authority",":path"]}}
creationTimestamp: "2021-03-03T22:19:56Z"
generation: 2
name: tracing
namespace: monitoring
resourceVersion: "791986493"
selfLink: /apis/getambassador.io/v2/namespaces/monitoring/tracingservices/tracing
uid: c69f7e0f-d621-4d52-b644-7bca965fba5f
spec:
ambassador_id:
- public
driver: zipkin
service: zipkin.monitoring:9411
tag_headers:
- :authority
- :path
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Issue Analytics
- State:
- Created a year ago
- Reactions:4
- Comments:5
Same issue here, has anyone in this thread managed to resolve it?
Edit: https://github.com/emissary-ingress/emissary/issues/4224#issuecomment-1172535547
Seems removing the V1 from CRD does the trick.
Same issue. Got a nice fat logs bill as a reward for upgrading emissary.