Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
kube_apiserver_metrics port is hardcoded to 6443
See original GitHub issueNote: If you have a feature request, you should contact support so the request can be properly tracked.
Output of the info page
2020-05-06 20:33:01 UTC | CORE | INFO | (pkg/util/log/log.go:482 in func1) | Found 'HTTP_PROXY' env var, using it for the Agent proxy settings
Getting the status from the agent.
2020-05-06 20:33:01 UTC | CORE | INFO | (pkg/util/log/log.go:482 in func1) | Found 'HTTPS_PROXY' env var, using it for the Agent proxy settings
2020-05-06 20:33:01 UTC | CORE | INFO | (pkg/util/log/log.go:482 in func1) | Found 'NO_PROXY' env var, using it for the Agent proxy settings
===============
Agent (v7.19.1)
===============
Status date: 2020-05-06 20:33:01.593464 UTC
Agent start: 2020-05-06 20:27:16.797371 UTC
Pid: 28831
Go Version: go1.13.8
Python Version: 3.8.1
Build arch: amd64
Check Runners: 4
Log Level: INFO
Paths
=====
Config File: /etc/datadog-agent/datadog.yaml
conf.d: /etc/datadog-agent/conf.d
checks.d: /etc/datadog-agent/checks.d
Clocks
======
NTP offset: -410µs
System UTC time: 2020-05-06 20:33:01.593464 UTC
Host Info
=========
bootTime: 2020-05-06 11:18:41.000000 UTC
kernelVersion: 4.14.173-137.229.amzn2.x86_64
os: linux
platform: debian
platformFamily: debian
platformVersion: bullseye/sid
procs: 252
uptime: 9h8m46s
Hostnames
=========
ec2-hostname: redacted
host_aliases: [redacted]
hostname: redacted
instance-id: redacted
socket-fqdn: datadog-99lsg
socket-hostname: datadog-99lsg
host tags:
cluster_name: redacted
aws_instance_type:m5.xlarge
kube_role:master
hostname provider: aws
unused hostname providers:
configuration/environment: hostname is empty
gce: unable to retrieve hostname from GCE: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname
Metadata
========
cloud_provider: AWS
hostname_source: aws
=========
Collector
=========
Running Checks
==============
cpu
---
Instance ID: cpu [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 6, Total: 126
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2020-05-06 20:32:49.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:49.000000 UTC
disk (2.8.0)
------------
Instance ID: disk:e5dffb8bef24336f [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 196, Total: 4,312
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 121ms
Last Execution Date : 2020-05-06 20:32:56.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:56.000000 UTC
docker
------
Instance ID: docker [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 556, Total: 12,232
Events: Last Run: 0, Total: 1
Service Checks: Last Run: 1, Total: 22
Average Execution Time : 227ms
Last Execution Date : 2020-05-06 20:32:48.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:48.000000 UTC
file_handle
-----------
Instance ID: file_handle [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 5, Total: 110
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2020-05-06 20:32:55.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:55.000000 UTC
io
--
Instance ID: io [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 78, Total: 1,662
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2020-05-06 20:32:47.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:47.000000 UTC
kube_apiserver_metrics (1.2.3)
------------------------------
Instance ID: kube_apiserver_metrics:47c7b885b8b976d7 [ERROR]
Configuration Source: file:/etc/datadog-agent/conf.d/kube_apiserver_metrics.d/auto_conf.yaml
Total Runs: 23
Metric Samples: Last Run: 0, Total: 0
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 1, Total: 23
Average Execution Time : 12ms
Last Execution Date : 2020-05-06 20:32:58.000000 UTC
Last Successful Execution Date : Never
Error: HTTPSConnectionPool(host='10.194.170.138', port=6443): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f83c7e243a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f83c7e243a0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.194.170.138', port=6443): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f83c7e243a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 820, in run
self.check(instance)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kube_apiserver_metrics/kube_apiserver_metrics.py", line 70, in check
self.process(self.kube_apiserver_config, metric_transformers=self.metric_transformers)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 474, in process
for metric in self.scrape_metrics(scraper_config):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 414, in scrape_metrics
response = self.poll(scraper_config)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 675, in poll
response = self.send_request(endpoint, scraper_config, headers)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 701, in send_request
return http_handler.get(endpoint, stream=True, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 283, in get
return self._request('get', url, options)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 325, in _request
return getattr(requests, method)(url, **new_options)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='10.194.170.138', port=6443): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f83c7e243a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
kube_controller_manager (1.6.1)
-------------------------------
Instance ID: kube_controller_manager:79f2168a712e06da [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/kube_controller_manager.d/auto_conf.yaml
Total Runs: 22
Metric Samples: Last Run: 1,023, Total: 22,506
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 2, Total: 44
Average Execution Time : 802ms
Last Execution Date : 2020-05-06 20:32:50.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:50.000000 UTC
kube_scheduler (1.3.1)
----------------------
Instance ID: kube_scheduler:75521d89b4fa93f3 [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/kube_scheduler.d/auto_conf.yaml
Total Runs: 22
Metric Samples: Last Run: 309, Total: 6,798
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 2, Total: 44
Average Execution Time : 223ms
Last Execution Date : 2020-05-06 20:32:57.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:57.000000 UTC
kubelet (4.0.0)
---------------
Instance ID: kubelet:d884b5186b651429 [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 483, Total: 10,611
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 4, Total: 88
Average Execution Time : 957ms
Last Execution Date : 2020-05-06 20:32:55.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:55.000000 UTC
load
----
Instance ID: load [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
Total Runs: 23
Metric Samples: Last Run: 6, Total: 138
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2020-05-06 20:33:01.000000 UTC
Last Successful Execution Date : 2020-05-06 20:33:01.000000 UTC
memory
------
Instance ID: memory [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 17, Total: 374
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2020-05-06 20:32:53.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:53.000000 UTC
network (1.15.1)
----------------
Instance ID: network:e0204ad63d43c949 [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
Total Runs: 23
Metric Samples: Last Run: 31, Total: 713
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 1ms
Last Execution Date : 2020-05-06 20:33:00.000000 UTC
Last Successful Execution Date : 2020-05-06 20:33:00.000000 UTC
ntp
---
Instance ID: ntp:b3b7f74889e8d254 [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/ntp.yaml
Total Runs: 1
Metric Samples: Last Run: 1, Total: 1
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 1, Total: 1
Average Execution Time : 8ms
Last Execution Date : 2020-05-06 20:27:28.000000 UTC
Last Successful Execution Date : 2020-05-06 20:27:28.000000 UTC
prometheus (3.2.1)
------------------
Instance ID: prometheus:fluent_bit:a70e3ef46344fd9a [OK]
Configuration Source: kubelet:docker://61e42875c6c33ff0fbf65bcc4836c460fdf31d682eb9d9fff91a1fb5ac6e89af
Total Runs: 23
Metric Samples: Last Run: 27, Total: 621
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 1, Total: 23
Average Execution Time : 11ms
Last Execution Date : 2020-05-06 20:32:59.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:59.000000 UTC
uptime
------
Instance ID: uptime [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
Total Runs: 22
Metric Samples: Last Run: 1, Total: 22
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2020-05-06 20:32:52.000000 UTC
Last Successful Execution Date : 2020-05-06 20:32:52.000000 UTC
========
JMXFetch
========
Initialized checks
==================
no checks
Failed checks
=============
no checks
=========
Forwarder
=========
Transactions
============
CheckRunsV1: 22
Connections: 0
Containers: 0
Dropped: 0
DroppedOnInput: 0
Events: 0
HostMetadata: 0
IntakeV1: 5
Metadata: 0
Pods: 0
Processes: 0
RTContainers: 0
RTProcesses: 0
Requeued: 0
Retried: 0
RetryQueueSize: 0
Series: 0
ServiceChecks: 0
SketchSeries: 0
Success: 49
TimeseriesV1: 22
API Keys status
===============
API key ending with 49904: API Key valid
==========
Endpoints
==========
https://app.datadoghq.com - API Key ending with:
- 49904
==========
Logs Agent
==========
Logs Agent is not running
============
System Probe
============
System Probe is not running:
Errors
======
error setting up remote system probe util, socket path does not exist: stat /opt/datadog-agent/run/sysprobe.sock: no such file or directory
=========
Aggregator
=========
Checks Metric Sample: 60,851
Dogstatsd Metric Sample: 40
Event: 2
Events Flushed: 2
Number Of Flushes: 22
Series Flushed: 51,299
Service Check: 558
Service Checks Flushed: 558
=========
DogStatsD
=========
Event Packets: 0
Event Parse Errors: 0
Metric Packets: 39
Metric Parse Errors: 0
Service Check Packets: 0
Service Check Parse Errors: 0
Udp Bytes: 2,007
Udp Packet Reading Errors: 0
Udp Packets: 40
Uds Bytes: 0
Uds Origin Detection Errors: 0
Uds Packet Reading Errors: 0
Uds Packets: 1
=====================
Datadog Cluster Agent
=====================
- Datadog Cluster Agent endpoint detected: https://100.66.249.12:5005
Successfully connected to the Datadog Cluster Agent.
- Running: 1.5.2+commit.60ee741
Additional environment details (Operating System, Cloud provider, etc): Kops AWS
Steps to reproduce the issue:
- Create a cluster using kops
- install the DD agent and cluster agent
Describe the results you received:
Autodiscovery for the kube_apiserver_metrics integration (that is automatically used) has a hardcoded port of 6443. Kops sets up the API server on 443. There is no way for me to change this in the DD configuration.
Describe the results you expected:
I would like a way to specify the API Server port - or better yet - have the kube_apiserver_metrics integration auto configure properly.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:6 (1 by maintainers)
Top Results From Across the Web
How to Monitor Kubernetes API Server - Sysdig
Just get access either through the kubernetes service HTTPs port (443), or any of the endpoints HTTPs port (6443). Let's see how you...
Read more >Kubernetes HA under X-ray - Heptio
A possible small variation of this approach is to bind the kube-apiserver to the secure address https://127.0.0.1:6443 , and to omit the -- ......
Read more >kube-apiserver - Kubernetes
Enables anonymous requests to the secure port of the API server. Requests that are not rejected by another authentication method are treated ...
Read more >Kubernetes Attack Surface | Optiv
6443 /TCP (Kubernetes API Port) 8443/TCP (Minikube API Port) 8080/TCP (Insecure K8s API Port) 10250/TCP (kubelet API)
Read more >Approaching Kubernetes Security - Splunk
Those processes are: kube-apiserver, kube-controller-manager and ... API transport security port: 6443; Authentication: Client Certificates, ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hi @michaelajr, thanks for opening up this issue!
You can replace the
auto_conf.yamlfor kubernetes_apiserver_metrics by mounting your own modified auto-configuration with the correct port. Here are some examples of different ways to achieve that, either mounting theconf.d/kubernetes_apiserver_metrics.d/directory or using a configmap and overriding in the manifest.@gcsabbagh I’m closing the issue but let us know if the solution explained above didn’t work for you.