Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

capture_exception raise TransportException and abort transaction

See original GitHub issue

Describe the bug: …

During an exception handling capture_exeption fail to send data and as result transaction is aborted. Seems the problem is decoding the data payload (memoryview type) to urllib3

This snippet replicate the issue in my env, just need to setup elasticapm client:

from shapely.geometry import Polygon
import geopandas as gpd
import elasticapm
import traceback

client = elasticapm.Client(<a config dictionary>)
client.begin_transaction(transaction_type="script")
result = 'unknown'
try:
    with elasticapm.capture_span(name="a_name", labels={}):
        a = Polygon()
        b = gpd.GeoDataFrame(geometry=[a])
        b.centroid.x[0]
    result = 'success'
except Exception:
    message = traceback.format_exc()
    client.capture_exception()
    result = 'failed'
finally:
    client.end_transaction(name="transaction_name", result=result)

result in the following trace:

2021-12-22 12:22:56,737 [urllib3.connectionpool] [DEBUG] https://<hide>:443 "POST /intake/v2/events HTTP/1.1" 400 None
2021-12-22 12:22:56,828 [elasticapm.transport] [ERROR] Failed to submit message: 'HTTP 400: {"accepted":24,"errors":[{"message":"decode error: data read error: v2.errorRoot.Error: v2.errorEvent.Exception: v2.errorException.Stacktrace: []v2.stacktraceFrame: v2.stacktraceFrame.Vars: Read: unexpected value type: 0, error found in #10 byte of ...|l_value\\": NaN}, \\"pre|..., bigger context ...|,\\\\n      dtype=object)\\", \\"op\\": \\"x\\", \\"null_value\\": NaN}, \\"pre_context\\": [\\"    # type: (str, np.array[|...","document":"{\\"error\\": {\\"context\\": {\\"custom\\": {}}, \\"id\\": \\".....\\", \\"exception\\": {\\"message\\": \\"IndexError: list index out of range\\", \\"type\\": \\"IndexError\\", \\"module\\": \\"builtins\\", \\"stacktrace\\": [{\\"abs_path\\": \\"./stripe_delivery_pipeline.py\\", \\"filename\\": \\"stripe_delivery_pipeline.py\\", \\"module\\": \\"__main__\\", \\"function\\": \\"main\\", \\"lineno\\": 1970, \\"library_frame\\": false, \\"vars\\": {\\"elasticapm_span_labels\\": null, \\"elasticapm_span_name\\": \\"delivery-pipeline-main\\", \\"location\\": \\"undefined\\", \\"country\\": \\"undefined\\", \\"message\\": \\"Traceback (most recent call last):\\\\n  File \\\\\\"./stripe_delivery_pipeline.py\\\\\\", line 1970, in main\\\\n    status, message, country, location = real_main_multi_AOI(scene_set_id, scene_id_tag)\\\\n  File \\\\\\"./stri...\\", \\"status\\": \\"Failed\\", \\"result\\": null, \\"labels\\": {\\"pod_name\\": null, \\"job_id\\": null, \\"sceneset_id\\": \\".........\\", \\"scene_id_tag\\": null}, \\"     ...[cut]

and

Traceback (most recent call last):
  File "/home/lpirelli/miniconda/envs/py36/lib/python3.6/site-packages/elasticapm/transport/base.py", line 239, in _flush
    self.send(data)
  File "/home/lpirelli/miniconda/envs/py36/lib/python3.6/site-packages/elasticapm/transport/http.py", line 113, in send
    raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.exceptions.TransportException: HTTP 400: {"accepted":24,"errors":[{"message":"decode error: data read error: v2.errorRoot.Error: v2.errorEvent.Exception: v2.errorException.Stacktrace: []v2.stacktraceFrame: v2.stacktraceFrame.Vars: Read: unexpected value type: 0, error found in #10 byte of ...|l_value\": NaN}, \"pre|..., bigger context ...|,\\n      dtype=object)\", \"op\": \"x\", \"null_value\": NaN}, \"pre_context\": [\"    # type: (str, np.array[|...","document":"{\"error\": {\"context\": {\"custom\": {}}, \"id\": \"....\", \"exception\": {\"message\": \"IndexError: list index out of range\", \"type\": \"IndexError\", \"module\": \"builtins\", \"stacktra    ....[cut]

data payload in memory view is:

a= b'\x1f\x8b\x08\x00a\x11\xc3a\x02\xff\xed}{w#\xb7\xb1\xe7\xff\xfb)xx\xb3\xeb\xf1\x1e\x91\xc2\xb3\x1f:q\xb2\xb2F\xb6\x95;\x0fE\x92\x938\xe39}\xd0\xddh\xa93$\x9b\xe9&53\xf1z?\xfbV\x01\xfd\xe4C\xa44\x92F\xf6pr\xaf%u\x03\xe8z\xa1\xf0.... [cut]'

str(a, 'utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Environment (please complete the following information)

OS: Linux
Python version:Python 3.6.10 |Anaconda, Inc.| (default, Mar 23 2020, 23:13:11) [GCC 7.3.0] on linux
APM Server version: elasticsearch==7.10.1, elastic-apm==6.4.0