Connection Refused at List - K8S Client using localhost:80 instead of API Server IP:PORT

Just deployed Kube-Promehteus-Stack 13.13.1 which includes grafana chart 6.4.* which deployed kiwigrid/k8s-sidecar:1.10.6. The init container boots up but fails due to a connection refused error:

[2021-03-02 21:50:50] Starting collector
[2021-03-02 21:50:50] No folder annotation was provided, defaulting to k8s-sidecar-target-directory
[2021-03-02 21:50:50] Selected resource type: ('secret', 'configmap')
[2021-03-02 21:50:50] Config for cluster api loaded...
[2021-03-02 21:50:50] Unique filenames will not be enforced.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f136d514670>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/sidecar.py", line 73, in <module>
    main()
  File "/app/sidecar.py", line 65, in main
    listResources(label, labelValue, targetFolder, url, method, payload,
  File "/app/resources.py", line 72, in listResources
    ret = getattr(v1, _list_namespaced[resource])(namespace=namespace, label_selector=labelSelector)
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15938, in list_namespaced_secret
    return self.list_namespaced_secret_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 16049, in list_namespaced_secret_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 239, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 212, in request
    r = self.pool_manager.request(method, url,
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 75, in request
    return self.request_encode_url(
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/namespaces/telemetry-system/secrets?labelSelector=grafana_datasource (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f136d514670>: Failed to establish a new connection: [Errno 111] Connection refused'))

Looking at the init container spec it only starts up with this configuration

    Environment:
      METHOD:           LIST
      LABEL:            grafana_datasource
      FOLDER:           /etc/grafana/provisioning/datasources
      RESOURCE:         both
      SKIP_TLS_VERIFY:  true
    Mounts:
      /etc/grafana/provisioning/datasources from sc-datasources-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-prometheus-stack-grafana-token-j2kzz (ro)

Unsure how to make it use the correct address where it seems like it should be using the in cluster connection since “Config for cluster api loaded” was logged.

[mitchellmaler]

Ah I see. So it is just using the “default” config instead of raising an exception like before which would have been caught to then try to load the in cluster config.

Either it needs to check the loaded config to see if it was set to the defaults (localhost) or require a flag to to switch between loaded vs in cluster config. The latter would be a breaking change so some new logic to just check would probably be best.

[mitchellmaler]

@jekkel That did the trick! Thanks!