v4.1.0 Introduces `http.client.RemoteDisconnected` error
See original GitHub issueDescribe the bug: … Events getting reported to APM-Server are randomly getting rejected. It appears that every so often the event getting reported is being met with a RemoteDisconnected exception.
To Reproduce
- Setup a django project as per directions
- Hit the homepage of the project
- Wait for several events to hit the APM server
- One will eventually cause the exception
Expected behavior: … No Exception
Environment (please complete the following information)
- OS: Mojave but running under
python:3
docker container - Python version: 3.7.2
- Framework and version: Django version 2.1.5
- APM Server version: apm-server version 6.6.0 (amd64), libbeat 6.6.0
- Agent version: 4.1.0
Additional context
downgrading to 4.0.3 remedies the issue.
Directly related: https://discuss.elastic.co/t/unable-to-reach-apm-server-connection-aborted-remotedisconnected-remote-end-closed-connection-without-response/166500
-
Agent config options
Click to expand
ELASTIC_APM = { 'SERVICE_NAME': 'web-backend', 'SECRET_TOKEN': '8149e743-2524-2524-2524-94aadd3fcde7', 'SERVER_URL': 'http://192.168.16.100:8200', 'DEBUG': True, }
-
requirements.txt
:Click to expand
django-allauth django-guardian django-qr-code django-icons django-phonenumber-field django-crispy-forms django-autocomplete-light django-money django-anymail django-debug-toolbar django-extensions django-countries djangorestframework psycopg2-binary stripe geoip2 redis celery sentry-sdk googlemaps django pillow phonenumbers gunicorn whitenoise elastic-apm
web_1 | Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://192.168.16.100:8200/intake/v2/events)"
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
web_1 | chunked=chunked)
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
web_1 | six.raise_from(e, None)
web_1 | File "<string>", line 2, in raise_from
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
web_1 | httplib_response = conn.getresponse()
web_1 | File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
web_1 | rv = real_getresponse(self, *args, **kwargs)
web_1 | File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
web_1 | response.begin()
web_1 | File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
web_1 | version, status, reason = self._read_status()
web_1 | File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
web_1 | raise RemoteDisconnected("Remote end closed connection without"
web_1 | http.client.RemoteDisconnected: Remote end closed connection without response
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/http.py", line 40, in send
web_1 | "POST", url, body=data, headers=self._headers, timeout=self._timeout, preload_content=False
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 323, in urlopen
web_1 | response = conn.urlopen(method, u.request_uri, **kw)
web_1 | File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
web_1 | return wrapped(*args, **kwargs)
web_1 | File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
web_1 | return wrapped(*args, **kwargs)
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
web_1 | _stacktrace=sys.exc_info()[2])
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
web_1 | raise six.reraise(type(error), error, _stacktrace)
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
web_1 | raise value.with_traceback(tb)
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
web_1 | chunked=chunked)
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
web_1 | six.raise_from(e, None)
web_1 | File "<string>", line 2, in raise_from
web_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
web_1 | httplib_response = conn.getresponse()
web_1 | File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
web_1 | rv = real_getresponse(self, *args, **kwargs)
web_1 | File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
web_1 | response.begin()
web_1 | File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
web_1 | version, status, reason = self._read_status()
web_1 | File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
web_1 | raise RemoteDisconnected("Remote end closed connection without"
web_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/base.py", line 184, in send_sync
web_1 | self.sync_transport.send(self, data)
web_1 | File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/http.py", line 53, in send
web_1 | raise TransportException(message, data, print_trace=print_trace)
web_1 | elasticapm.transport.base.TransportException: Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://192.168.16.100:8200/intake/v2/events)
Issue Analytics
- State:
- Created 5 years ago
- Reactions:5
- Comments:53 (16 by maintainers)
Top Results From Across the Web
http.client.RemoteDisconnected error while reading/parsing a ...
I am working on a simple url parser: the idea is to take a url in one column, attempt to resolve it and...
Read more >http.client.remotedisconnected - You.com | The AI Search ...
RemoteDisconnected: Remote end closed connection without response error using driver.quit() of ... elastic/apm-agent-pythonv4.1.0 Introduces `http.client.
Read more >Google searches seem dated.. my neo4j browser now on 7687
http.client.RemoteDisconnected: Remote end closed connection without response. During handling of the above exception, another exception ...
Read more >ProxyError: HTTPSConnectionPool : Forums - PythonAnywhere
I have the same error raise ProxyError(e, request=request) requests.exceptions.ProxyError: HTTPSConnectionPool(host='coronavirus-monitor.ru' ...
Read more >Application Programmer Interface (API) Reference Guide - Cisco
This guide introduces you to the Application Programmer ... The Cisco Proximity clients for smartphones and tablets ... Example 4: FECC Action request....
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@ciarancourtney @MrBones757 sorry to bug you folks again. I’ve implemented another approach that might work: https://github.com/elastic/apm-agent-python/pull/426. Specifically, in this branch, we add some fuzziness to the time between requests to the APM Server (between 90-110% of the configured value). The original reason I implemented this was to avoid stampedes when multiple workers are booted up at the same time, but it might help with this race condition as well.
I started to look at APM server a little bit closely and tried version 7.0.0-alpha2 and tried to find something relevant in server debug logs but got nothing valuable so went back to 6.6.0.
Then I found this setting: https://www.elastic.co/guide/en/apm/server/current/configuration-process.html#read_timeout When changed from default 30s to:
read_timeout: 45s
I’m now unable to reproduce the issue with my script. Also there are no more FIN packets from the server in wireshark. Even those not harmful between HTTP requests.I also noticed that the 30s may very well be racing with 30000ms from here: https://github.com/elastic/apm-agent-python/blob/v4.1.0/elasticapm/conf/__init__.py#L223
Also changing it to 20000ms makes the issue disappear even if read_timeout on the server is still 30s. Not sure if changing it is desired though.
So maybe to workaround this issue either of these setting need to be changed so that the server read_timeout is just longer than agent metrics_interval by changing either of these settings. I’m not sure if this is a proper fix but much better than the current situation.
As for reproduction on your side maybe using the docker-compose.yml for the apm server I posted above is required? It’s also somehow a little bit harder to reproduce if apm server debug logging is enabled.
Also if there is some ticket on apm-server side could you please link it here?