ElasticAPM crash because of a missing context since version 6.3.0
See original GitHub issueDescribe the bug:
Since ElasticAPM release 6.3.0 our application has started to see a ton of crashes that we believed are cause by a recent change in ElasticAPM. The elasticsearch instrumentation package is trying to assign the type of DB to elasticsearch into an None context which causes it to crash.
Here’s a snippet of a strack trace from our app:
...
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticsearch/client/__init__.py", line 1026, in get
"GET", _make_path(index, doc_type, id), params=params, headers=headers
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 210, in call_if_sampling
return self.call(module, method, wrapped, instance, args, kwargs)
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/elasticsearch.py", line 113, in call
result_data = wrapped(*args, **kwargs)
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticsearch/transport.py", line 388, in perform_request
timeout=timeout,
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 210, in call_if_sampling
return self.call(module, method, wrapped, instance, args, kwargs)
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/elasticsearch.py", line 56, in call
self._update_context_by_request_data(span.context, instance, args, kwargs)
File "/var/lib/assemblyline/.local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/elasticsearch.py", line 72, in _update_context_by_request_data
context["db"] = {"type": "elasticsearch"}
TypeError: 'NoneType' object does not support item assignment
I believe this crash is related to a recent change in this commit: https://github.com/elastic/apm-agent-python/commit/ee75cb8f238303b79d0f697a7f2eca547a1dfe8c#diff-c8fb731f92134757656c157f5c3175bcb62e131c1fed1aec5041367603c204d0L62
You can see here, the context was previously assigned it’s DB type in a way where even if the context was None it would still work but now it assumes the context is a dictionary. I’m not creating a PR to fix this because I’m not 100% sure if the old way was changed for a reason.
Possible fix I have very limited understanding on what that context should be before reaching this function but possible fixes include:
-
Revert to the old way of assigning the DB type.
-
To test for None context before assigning a type.
if context is None:
context = {}
- or make sure span.context default value is an empty dict instead of None
To Reproduce
I have no easy way to reproduce this crash because it does not happen all the time.
Environment (please complete the following information)
- OS: Linux
- Python version: 3.7.11
- Framework and version [e.g. Django 2.1]: Flask 2.0.1
- APM Server version: 7.12
- Agent version: 6.3.0+
Aditional Information
Our app in launched in Gunicorn using gevent workers.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:2
- Comments:5 (2 by maintainers)
@cccs-sgaron thanks for opening the issue! I went through the code, and the only possible way that I currently see how this could happen is if the Elasticsearch span is a
DroppedSpan
. Dropped spans happen in two situations:transaction_max_spans
I opened #1190 to address the issue. The test case in that pull request triggers the same stack trace as the one you added to this issue.
@cccs-sgaron @esseti Awesome, thanks for the feedback. I just pushed version 6.3.3 of the agent with the fix from #1190