Capturing and logging stdout/stderr on workers

For various reasons, it is nice to be able to use stdout and stderr on workers and see the results. This can be useful in simplistic debugging attempts, using 3rd party code that already prints to one (or both) of these, etc. However it appears stdout and stderr are ignored in these settings currently.

A very useful feature would be to capture stdout and stderr on the worker and redirect them to a logger. One simple option would be to use the loggers distributed.worker.stdout and distributed.worker.stderr with info level messages. These loggers seem to be unused now and redirecting to them seems to work fine. This would at least give users one reasonable way to get this information.

[fjetter]

In https://github.com/dask/distributed/pull/5217 (the ticket is a bit cryptic, below example pseudo code) we built a functionality to support “remote printing” that may be useful for some people in here. It appears we never updated the docs appropriately, at least I couldn’t find it.

from distributed import Client
from distributed.worker import print, warn

def func():
    print("foo")
c = Client("<ip_to_remote_scheduler>")
c.submit(func)

If you run a function on a remote worker and are using the distributed print function, it will print this message on stdout of your worker and forwards the message to the client and prints it on the client as well. The same works for warn as the stdlib warn works and you can build your own client handlers using Client.subscribe_topic in combination with Worker.log_event.

This obviously only scales to a certain level as was already pointed out with collecting logging information in a central place. At the very least the “simplistic debugging attempt” use case should be covered by this. For proper logging, I would not recommend using this.

[maxbane]

@ntonasa See my gist here for a proof-of-concept of how to use the subscribe mechanism that was added in #5217 to configure the forwarding of arbitrary logging statements by tasks running on workers to the client session.