Include exception type and reason in structured logging json

Currently when structured logging is enabled (using either the JsonFormatter or TerseJsonFormatter), logger.exception lines don’t include a traceback nor any exception information at all (other than what’s presented in the string passed to logger.exception.

This is due to us ignoring the exc_text and exc_info attributes of the log record when during formatting:

https://github.com/matrix-org/synapse/blob/18edc9ab06d8ed07c1cac918057226fad18030ce/synapse/logging/_terse_json.py#L23-L31

While it’s not quite feasible to include full tracebacks in structured logging (one should use a tool like Sentry.io for that), it may be useful to include the exception type ((i.e psycopg2.OperationalError) and text (i.e FATAL: sorry, too many clients already) as structured fields. This will give sysadmins a clue as to what might be going wrong, whereas Sentry is useful for pointing out the line where the issue occurred (and thus where developers may be able to go in and fix the issue).

[anoadragon453]

The logger call in question was this one:

https://github.com/matrix-org/synapse/blob/e9f2ad86034d27068941379f678e19bf280ed308/synapse/metrics/background_process_metrics.py#L216-L219

Which resulted in:

{
  "log": "Background process 'start_pushers' threw an exception",
  "namespace": "synapse.metrics.background_process_metrics",
  "level": "ERROR",
  "time": 1623655604.36,
  "request": "start_pushers-0",
  "server_name": "example.com"
}

Which doesn’t seem to include any useful exception information… unless it’s getting lost in this specific instance?

[bradtgmurray]

Also an issue with request handling failures:

This code block here: https://github.com/matrix-org/synapse/blob/4b965c862dc66c0da5d3240add70e9b5f0aa720b/synapse/http/server.py#L93-L98

precise output:

2021-07-09 15:20:37,478 - synapse.http.server - 93 - ERROR - PUT-6 - Failed handle request via 'RoomTypingRestServlet': <SynapseRequest at 0x7f8b33668640 method='PUT' uri='<redacted by brad for github>' clientproto='HTTP/1.1' site=8083>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render
    callback_return = await raw_callback_return
  File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/v1/room.py", line 1161, in on_PUT
    raise Exception("Got /typing request on instance that is not typing writer")
Exception: Got /typing request on instance that is not typing writer

structured output:

{"log":"Failed handle request via 'RoomTypingRestServlet': <SynapseRequest at 0x7f257b020280 method='PUT' uri='<redacted by brad for github>' clientproto='HTTP/1.1' site=8083>","namespace":"synapse.http.server","level":"ERROR","time":1625844440.35,"request":"PUT-15","ip_address":"::ffff:192.168.39.92","site_tag":8083,"requester":"<redacted by brad for github>","authenticated_entity":"@bradtest:beeper-brad.com","method":"PUT","url":"<redacted by brad for github>","protocol":"HTTP/1.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36","server_name":"<redacted by brad for github>"}

without a backtrace