Proposal: highlight exact operation in tracebacks

Hi! I recently created executing, a library which can identify the AST node being executed by a frame in many cases. I’d like to use this to improve Python tracebacks in many places, starting with IPython. It’s always disappointed me that Python only points to a line in each frame in tracebacks, especially when the expression/statement at fault spans many lines. Sometimes it can be really hard to interpret.

This toy library offers a nice example of what tracebacks using executing can look like:

Basically, highlighting the text background could make it clear what’s being executed without interfering with other coloring and formatting such as syntax highlighting.

I tried to jump to coding this straight away but quickly saw that it’s quite complicated. The introspection magic stuff is already done, it’s just a matter of updating the formatting in ultratb, but that still looks quite hard, especially as I’m new to this code. So I’m creating this issue first to check if the idea is acceptable before wasting my time.

I’m expecting two main concerns: reliability and dependencies.

executing is extremely reliable. By that I mean that I’m confident it will never identify the wrong node. If the linked explanation doesn’t convince you, I encourage you to try finding a case where it goes wrong. If it helps, I’m also using executing in two other libraries of mine: sorcery and snoop. However executing often can’t identify the node at all, so not all frames will have something highlighted. Over time this situation will improve as I implement more operations.

Two new dependencies would need to be added: executing and asttokens (to convert AST nodes to source code positions, I don’t know any alternative). The only transitive dependency from these is six. In my experience asttokens is an excellent library with active and responsive contributors.

Does this sound good to you?

[Carreau]

Ultratb is a real mess and really old. Any refactor to it is welcomed; I’ve been willing from some time to separate the traceback information extracting from the formatting, to potentially make a nice HTML repr…

I don’t have any objections to add dependencies; we might want to make sure they are packaged in conda/debian and some of our downstream redistributor and if not asses the difficulty this would add for them. (asttokens already available, and six as well)

@scopatz and @gforsyth might be interested in this for xonsh. If we want/can extract these into common packages I’m all for it.

There is also https://github.com/qix-/better-exceptions that looks nice; though showing variable values can be a security issue s can’t be enabled by default.

If that gets in, It will probably be for an 8.0 release.

[asmeurer]

Huh, maybe it’s just coincidental timing. What conversation on Twitter led you here?

https://twitter.com/Mbussonn/status/1227386489987723264

I don’t want to hijack this conversation any more. I’ll open an issue on the stack_data repo.