ProtocolError - InvalidChunkLength
See original GitHub issueA quay.io/kiwigrid/k8s-sidecar:1.21.0
container deployed alongside Grafana in a k8s 1.24 evaluation cluster seems to start correctly, but afterwards floods logs with ProtocolError
messages as seen below:
{"time": "2022-11-19T12:12:04.355037+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2022-11-19T12:12:04.355222+00:00", "level": "WARNING", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2022-11-19T12:12:04.355347+00:00", "level": "INFO", "msg": "Loading incluster config ..."}
{"time": "2022-11-19T12:12:04.356003+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://api.eval.420214.internal.prod.<redacted>:443' loaded..."}
{"time": "2022-11-19T12:12:04.356101+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2022-11-19T12:12:04.356173+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}
{"time": "2022-11-19T12:12:08.031424+00:00", "level": "INFO", "msg": "Subprocess exiting gracefully"}
{"time": "2022-11-19T12:12:08.032719+00:00", "level": "INFO", "msg": "Subprocess exiting gracefully"}
{"time": "2022-11-19T12:12:08.032763+00:00", "level": "INFO", "msg": "Subprocess exiting gracefully"}
{"time": "2022-11-19T12:12:59.386760+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
{"time": "2022-11-19T12:12:59.387126+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
{"time": "2022-11-19T12:13:54.412196+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
{"time": "2022-11-19T12:13:54.426154+00:00", "level": "ERROR", "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))"}
...
Is this a known issue?
Issue Analytics
- State:
- Created 10 months ago
- Comments:5 (1 by maintainers)
Top Results From Across the Web
Exceptions - urllib3 1.26.13 documentation
Renamed to ProtocolError but aliased for backwards compatibility. exception urllib3.exceptions.DecodeError# ... Invalid chunk length in a chunked response.
Read more >requests - ChunkedEncodingError with requests.get, but ...
I think the reason for this error is server is not correctly encoding the chunk. some of the chunk sizes are not integer...
Read more >Getting exception in _update_chunk_length · Issue #1516
ValueError is less than ideal to indicate a protocol error. In my case and issue 4248 above, I'm going to guess that the...
Read more >requests.exceptions.ChunkedEncodingError: - Twitter API v2
Hi there, I've been using the code sample your team provides on GitHub (which I believe you've shared above) and get the same...
Read more >Solved: Error with python script to fetch devices from NSO
ProtocolError : ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
And this is a stack trace which I managed to retrieve.
The code was used (it was running in a loop because most of the time it works as expected):
So far it looks like some networking issues rather than an issue with k8s-sidecar itself.
Related issues in the Kubernetes python client:
The suggested solution (which is actually a workaround) is to implement a retry logic. However, looking at the k8s-sidecar’s code it seems to be not really needed, because the retry logic is implemented on a higher level already. Personally, I’d decrease the severity of the corresponding log message from ERROR to WARNING, because, in fact, everything is working fine.