ProtocolError - InvalidChunkLength

A quay.io/kiwigrid/k8s-sidecar:1.21.0 container deployed alongside Grafana in a k8s 1.24 evaluation cluster seems to start correctly, but afterwards floods logs with ProtocolError messages as seen below:

{"time": "2022-11-19T12:12:04.355037+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2022-11-19T12:12:04.355222+00:00", "level": "WARNING", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2022-11-19T12:12:04.355347+00:00", "level": "INFO", "msg": "Loading incluster config ..."}
{"time": "2022-11-19T12:12:04.356003+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://api.eval.420214.internal.prod.<redacted>:443' loaded..."}
{"time": "2022-11-19T12:12:04.356101+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2022-11-19T12:12:04.356173+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}
{"time": "2022-11-19T12:12:08.031424+00:00", "level": "INFO", "msg": "Subprocess exiting gracefully"}
{"time": "2022-11-19T12:12:08.032719+00:00", "level": "INFO", "msg": "Subprocess exiting gracefully"}
{"time": "2022-11-19T12:12:08.032763+00:00", "level": "INFO", "msg": "Subprocess exiting gracefully"}
{"time": "2022-11-19T12:12:59.386760+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
{"time": "2022-11-19T12:12:59.387126+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
{"time": "2022-11-19T12:13:54.412196+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
{"time": "2022-11-19T12:13:54.426154+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
...

Is this a known issue?

[Zebradil]

And this is a stack trace which I managed to retrieve.

Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 441, in _error_catcher
    yield
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 767, in read_chunked
    self._update_chunk_length()
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 704, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/dbg.py", line 6, in <module>
    for event in stream:
  File "/app/.venv/lib/python3.10/site-packages/kubernetes/watch/watch.py", line 165, in stream
    for line in iter_resp_lines(resp):
  File "/app/.venv/lib/python3.10/site-packages/kubernetes/watch/watch.py", line 56, in iter_resp_lines
    for seg in resp.stream(amt=None, decode_content=False):
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 575, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 755, in read_chunked
    with self._error_catcher():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/.venv/lib/python3.10/site-packages/urllib3/response.py", line 458, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

The code was used (it was running in a loop because most of the time it works as expected):

from kubernetes import client, config, watch

config.load_incluster_config()
v1 = client.CoreV1Api()
stream = watch.Watch().stream(v1.list_config_map_for_all_namespaces, label_selector='grafana_dashboard=1', timeout_seconds=60, _request_timeout=66)
for event in stream:
    print(event['object'].metadata.name)

So far it looks like some networking issues rather than an issue with k8s-sidecar itself.

[Zebradil]

Related issues in the Kubernetes python client:

• https://github.com/kubernetes-client/python/issues/728
• https://github.com/kubernetes-client/python/issues/1693

The suggested solution (which is actually a workaround) is to implement a retry logic. However, looking at the k8s-sidecar’s code it seems to be not really needed, because the retry logic is implemented on a higher level already. Personally, I’d decrease the severity of the corresponding log message from ERROR to WARNING, because, in fact, everything is working fine.