Kube-State-Metrics are not processed (MetricBeat)
See original GitHub issueChart version: 7.3.0
Kubernetes version: 1.12.8
Kubernetes provider: Azure
Helm Version:
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
helm get release
output
REVISION: 1
RELEASED: Mon Sep 16 15:09:44 2019
CHART: metricbeat-7.3.0
USER-SUPPLIED VALUES:
{}
COMPUTED VALUES:
extraEnvs: []
extraVolumeMounts: []
extraVolumes: []
fullnameOverride: ""
hostPathRoot: /var/lib
image: docker.elastic.co/beats/metricbeat
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.3.0
kube-state-metrics:
affinity: {}
collectors:
certificatesigningrequests: true
configmaps: true
cronjobs: true
daemonsets: true
deployments: true
endpoints: true
horizontalpodautoscalers: true
ingresses: true
jobs: true
limitranges: true
namespaces: true
nodes: true
persistentvolumeclaims: true
persistentvolumes: true
poddisruptionbudgets: true
pods: true
replicasets: true
replicationcontrollers: true
resourcequotas: true
secrets: true
services: true
statefulsets: true
global: {}
hostNetwork: false
image:
pullPolicy: IfNotPresent
repository: quay.io/coreos/kube-state-metrics
tag: v1.6.0
nodeSelector: {}
podAnnotations: {}
podSecurityPolicy:
annotations: {}
enabled: false
prometheus:
monitor:
additionalLabels: {}
enabled: false
namespace: ""
prometheusScrape: true
rbac:
create: true
replicas: 1
securityContext:
enabled: true
fsGroup: 65534
runAsUser: 65534
service:
loadBalancerIP: ""
nodePort: 0
port: 8080
type: ClusterIP
serviceAccount:
create: true
imagePullSecrets: []
name: null
tolerations: []
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
managedServiceAccount: true
metricbeatConfig:
kube-state-metrics-metricbeat.yml: |
metricbeat.modules:
- module: kubernetes
enabled: true
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
period: 10s
hosts: ["${KUBE_STATE_METRICS_HOSTS:kube-state-metrics:8080}"]
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
metricbeat.yml: |
system:
hostfs: /hostfs
metricbeat.modules:
- module: kubernetes
metricsets:
- container
- node
- pod
- system
- volume
period: 10s
hosts: ["localhost:10255"]
processors:
- add_kubernetes_metadata:
in_cluster: true
- module: kubernetes
enabled: true
metricsets:
- event
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
processes: ['.*']
process.include_top_n:
by_cpu: 5
by_memory: 5
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
nameOverride: ""
podAnnotations: {}
podSecurityContext:
privileged: false
runAsUser: 0
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
replicas: 1
resources:
limits:
cpu: 1000m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
secretMounts: []
serviceAccount: ""
terminationGracePeriod: 30
tolerations: []
updateStrategy: RollingUpdate
HOOKS:
MANIFEST:
---
# Source: metricbeat/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-metricbeat-config
labels:
app: "metricbeat-metricbeat"
chart: "metricbeat-7.3.0"
heritage: "Tiller"
release: "metricbeat"
data:
kube-state-metrics-metricbeat.yml: |
metricbeat.modules:
- module: kubernetes
enabled: true
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
period: 10s
hosts: ["${KUBE_STATE_METRICS_HOSTS:kube-state-metrics:8080}"]
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
metricbeat.yml: |
system:
hostfs: /hostfs
metricbeat.modules:
- module: kubernetes
metricsets:
- container
- node
- pod
- system
- volume
period: 10s
hosts: ["localhost:10255"]
processors:
- add_kubernetes_metadata:
in_cluster: true
- module: kubernetes
enabled: true
metricsets:
- event
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
processes: ['.*']
process.include_top_n:
by_cpu: 5
by_memory: 5
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
---
# Source: metricbeat/charts/kube-state-metrics/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: kube-state-metrics
chart: kube-state-metrics-1.6.0
heritage: Tiller
release: metricbeat
name: metricbeat-kube-state-metrics
imagePullSecrets:
[]
---
# Source: metricbeat/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat-metricbeat
labels:
app: "metricbeat-metricbeat"
chart: "metricbeat-7.3.0"
heritage: "Tiller"
release: "metricbeat"
---
# Source: metricbeat/charts/kube-state-metrics/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
labels:
app: kube-state-metrics
chart: kube-state-metrics-1.6.0
heritage: Tiller
release: metricbeat
name: metricbeat-kube-state-metrics
rules:
- apiGroups: ["certificates.k8s.io"]
resources:
- certificatesigningrequests
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources:
- cronjobs
verbs: ["list", "watch"]
- apiGroups: ["extensions", "apps"]
resources:
- daemonsets
verbs: ["list", "watch"]
- apiGroups: ["extensions", "apps"]
resources:
- deployments
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- endpoints
verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
resources:
- horizontalpodautoscalers
verbs: ["list", "watch"]
- apiGroups: ["extensions"]
resources:
- ingresses
verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources:
- jobs
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- limitranges
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- namespaces
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- nodes
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- persistentvolumeclaims
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- persistentvolumes
verbs: ["list", "watch"]
- apiGroups: ["policy"]
resources:
- poddisruptionbudgets
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- pods
verbs: ["list", "watch"]
- apiGroups: ["extensions", "apps"]
resources:
- replicasets
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- replicationcontrollers
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- resourcequotas
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- secrets
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- services
verbs: ["list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
verbs: ["list", "watch"]
---
# Source: metricbeat/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: metricbeat-metricbeat-cluster-role
labels:
app: "metricbeat-metricbeat"
chart: "metricbeat-7.3.0"
heritage: "Tiller"
release: "metricbeat"
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
verbs:
- get
- list
- watch
---
# Source: metricbeat/charts/kube-state-metrics/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
labels:
app: kube-state-metrics
chart: kube-state-metrics-1.6.0
heritage: Tiller
release: metricbeat
name: metricbeat-kube-state-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metricbeat-kube-state-metrics
subjects:
- kind: ServiceAccount
name: metricbeat-kube-state-metrics
namespace: production
---
# Source: metricbeat/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: metricbeat-metricbeat-cluster-role-binding
labels:
app: "metricbeat-metricbeat"
chart: "metricbeat-7.3.0"
heritage: "Tiller"
release: "metricbeat"
roleRef:
kind: ClusterRole
name: metricbeat-metricbeat-cluster-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: metricbeat-metricbeat
namespace: production
---
# Source: metricbeat/charts/kube-state-metrics/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: metricbeat-kube-state-metrics
labels:
app: kube-state-metrics
chart: "kube-state-metrics-1.6.0"
release: "metricbeat"
heritage: "Tiller"
annotations:
prometheus.io/scrape: 'true'
spec:
type: "ClusterIP"
ports:
- name: "http"
protocol: TCP
port: 8080
targetPort: 8080
selector:
app: kube-state-metrics
release: metricbeat
---
# Source: metricbeat/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: metricbeat-metricbeat
labels:
app: "metricbeat-metricbeat"
chart: "metricbeat-7.3.0"
heritage: "Tiller"
release: "metricbeat"
spec:
selector:
matchLabels:
app: "metricbeat-metricbeat"
release: "metricbeat"
updateStrategy:
type: RollingUpdate
template:
metadata:
annotations:
configChecksum: 85527e805efa1e96c9bc6516dcd39d3705e552817f5f57f32f2651a651eeb10
name: "metricbeat-metricbeat"
labels:
app: "metricbeat-metricbeat"
chart: "metricbeat-7.3.0"
heritage: "Tiller"
release: "metricbeat"
spec:
serviceAccountName: metricbeat-metricbeat
terminationGracePeriodSeconds: 30
volumes:
- name: metricbeat-config
configMap:
defaultMode: 0600
name: metricbeat-metricbeat-config
- name: data
hostPath:
path: /var/lib/metricbeat-metricbeat-production-data
type: DirectoryOrCreate
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varrundockersock
hostPath:
path: /var/run/docker.sock
containers:
- name: "metricbeat"
image: "docker.elastic.co/beats/metricbeat:7.3.0"
imagePullPolicy: "IfNotPresent"
args:
- "-e"
- "-E"
- "http.enabled=true"
livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
metricbeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: 1000m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
privileged: false
runAsUser: 0
volumeMounts:
- name: metricbeat-config
mountPath: /usr/share/metricbeat/kube-state-metrics-metricbeat.yml
readOnly: true
subPath: kube-state-metrics-metricbeat.yml
- name: metricbeat-config
mountPath: /usr/share/metricbeat/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: data
mountPath: /usr/share/metricbeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
# Necessary when using autodiscovery; avoid mounting it otherwise
# See: https://www.elastic.co/guide/en/beats/metricbeat/master/configuration-autodiscover.html
- name: varrundockersock
mountPath: /var/run/docker.sock
readOnly: true
---
# Source: metricbeat/charts/kube-state-metrics/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: metricbeat-kube-state-metrics
labels:
app: kube-state-metrics
chart: "kube-state-metrics-1.6.0"
release: "metricbeat"
heritage: "Tiller"
spec:
selector:
matchLabels:
app: kube-state-metrics
replicas: 1
template:
metadata:
labels:
app: kube-state-metrics
release: "metricbeat"
spec:
hostNetwork: false
serviceAccountName: metricbeat-kube-state-metrics
securityContext:
fsGroup: 65534
runAsUser: 65534
containers:
- name: kube-state-metrics
args:
- --collectors=certificatesigningrequests
- --collectors=configmaps
- --collectors=cronjobs
- --collectors=daemonsets
- --collectors=deployments
- --collectors=endpoints
- --collectors=horizontalpodautoscalers
- --collectors=ingresses
- --collectors=jobs
- --collectors=limitranges
- --collectors=namespaces
- --collectors=nodes
- --collectors=persistentvolumeclaims
- --collectors=persistentvolumes
- --collectors=poddisruptionbudgets
- --collectors=pods
- --collectors=replicasets
- --collectors=replicationcontrollers
- --collectors=resourcequotas
- --collectors=secrets
- --collectors=services
- --collectors=statefulsets
imagePullPolicy: IfNotPresent
image: "quay.io/coreos/kube-state-metrics:v1.6.0"
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 5
resources:
null
---
# Source: metricbeat/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: 'metricbeat-metricbeat-metrics'
labels:
app: 'metricbeat-metricbeat-metrics'
chart: 'metricbeat-7.3.0'
heritage: 'Tiller'
release: 'metricbeat'
spec:
replicas: 1
selector:
matchLabels:
app: 'metricbeat-metricbeat-metrics'
chart: 'metricbeat-7.3.0'
heritage: 'Tiller'
release: 'metricbeat'
template:
metadata:
annotations:
configChecksum: 85527e805efa1e96c9bc6516dcd39d3705e552817f5f57f32f2651a651eeb10
labels:
app: 'metricbeat-metricbeat-metrics'
chart: 'metricbeat-7.3.0'
heritage: 'Tiller'
release: 'metricbeat'
spec:
serviceAccountName: metricbeat-metricbeat
terminationGracePeriodSeconds: 30
volumes:
- name: metricbeat-config
configMap:
defaultMode: 0600
name: metricbeat-metricbeat-config
containers:
- name: "metricbeat"
image: "docker.elastic.co/beats/metricbeat:7.3.0"
imagePullPolicy: "IfNotPresent"
args:
- "-c"
- "/usr/share/metricbeat/kube-state-metrics-metricbeat.yml"
- "-e"
- "-E"
- "http.enabled=true"
livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
metricbeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: 1000m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBE_STATE_METRICS_HOSTS
value: "$(METRICBEAT_KUBE_STATE_METRICS_SERVICE_HOST):$(METRICBEAT_KUBE_STATE_METRICS_SERVICE_PORT_HTTP)"
securityContext:
privileged: false
runAsUser: 0
volumeMounts:
- name: metricbeat-config
mountPath: /usr/share/metricbeat/kube-state-metrics-metricbeat.yml
readOnly: true
subPath: kube-state-metrics-metricbeat.yml
- name: metricbeat-config
mountPath: /usr/share/metricbeat/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
Describe the bug:
The kube state metrics service created by the Metricbeat chart is named metricbeat-kube-state-metrics:8080
, but values.yaml mentions kube-state-metrics:8080
.
https://github.com/elastic/helm-charts/blob/master/metricbeat/values.yaml#L61
Expected behavior:
Processing kube stat metrics should work out of the box
Issue Analytics
- State:
- Created 4 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Metricbeat missing kube-state-metrics - Beats - Elastic Discuss
Filebeat works fine. I can see kubernetes.. and system.. in the Metricbeat data but do not see kube-state-metrics data and cannot figure out ......
Read more >metricbeat 7.6.2 · elastic/elastic - Artifact Hub
This is caused by the update of kube-state-metrics chart dependency which is renaming some labels in helm/charts#15261. The workaround is to use --force ......
Read more >lookup kube-state-metrics on IP:53 no such host - Stack ...
Make sure you have the Kube-State-Metrics deployed in your cluster in the kube-system namespace to make this work. Metricbeat will not come ...
Read more >Use Elastic Stack to implement integrated monitoring for ...
If the Metricbeat and kube-state-metrics services are deployed in the same ... Otherwise, data may not be displayed in the Kibana console.
Read more >Monitoring Kubernetes Cluster using Elastic Stack ... - Medium
Metricbeat will collect metrics from Kubernetes Cluster and… ... way so do not worry/panic if you see the password in plain YAML ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Latest released version has just gone out. Can you try again to reproduce it?
Confirmed! Metrics are coming in. Thanks a lot!