Allow pipelines log lines to include structured data

Use Case

It’s possible this is a bug and not a feature request. I’ve read a bunch of the Dagster code and read through the documentation and examples but am not finding an answer one way or another on whether or not this is possible today.

In the Dagit pipeline execution UI there is a panel for logging. Some rows in that panel are EVENT types. These sometimes seem to have a really nice sub-table of key/value pairs (so called “structured fields”). Here’s an example of one:

What I’m looking for is a way to get this same UI but for log lines produced by my own code in my pipeline/solid/etc. For example, I want to have a log line like this:

context.log.info("Did some really important task", task_name="foo", bar="baz")

And have a log line show up in the Dagit log panel that has a table with the task_name and bar keys and their values.

I see that in Dagit’s logger manager code there is some plumbing to pass through **kwargs quite deep but I haven’t read enough to really understand if I should expect the kwargs to make it out to a logger implementation and eventually show up in the Dagit UI.

Ideas of Implementation

Not sure yet, mostly this is more of a question issue to help understand if this is a feature that exists and what I might be doing wrong!

---

Message from the maintainers:

Excited about this feature? Give it a 👍. We factor engagement into prioritization.

[lamielle]

If what I’m looking for IS possible, I guess this Issue can be turned into a documentation improvement request to help explain how such a behavior can be configured!

[lamielle]

Thanks @catherinewu, @yuhan, and @schrockn. I’ll review the Asset Materialization and Output functionality in more detail, but on first glance I agree with @schrockn: really what I’m looking for is just some structure logging with structured output (that appears generally possible already in the UI log stream for EVENT entries).

One more note related to this: neither dagster.loggers.json_console_logger nor dagster.loggers.colored_console_logger seem to do anything when I pass values through kwargs of a logging call. I just see the message string and nothing else. Is this expected? Is it up to the logger implementation itself to do something with those values?

Maybe the approach here is to implement a customer logger that does… <something> with those kwargs values to expose them in the log stream UI? I’m not sure what <something> needs to be though.