Capturing stdout with slurm

We’ve been running our experiments with the slurm scheduler which captures stdout and stdin and pipes them to a file. This seems to confuse sacred, which doesn’t capture stdout in this case.

Easiest solution by far would be just telling sacred where to look for stdout. Is there a way to do that?

Thanks! A

[Qwlouse]

Ohh, I found the problem: Sys-based capturing relies on replacing sys.stdout and sys.stderr with custom wrappers. But the StreamHandler still maintains a copy of the old sys.stderr and logs to that. So in fact any Sacred experiment that uses capture mode sys looses their logging. That is clearly a problem!

You might be able to work around it with something like this, which should give you most logs (except for the “experiment started” logs):

@ex.pre_run_hook
def set_logger_stream(_run):
    _run.root_logger.handlers[0].stream = sys.stderr

I have to think about how to best fix this problem in a clean way. Let me know if you have any thoughts on this, especially also concerning the general integration of sacred with logging, since I am not entirely happy with the way it is currently done.

[Qwlouse]

That is an interesting case I hadn’t thought about. So far this is not possible, no. If you are fine with only capturing stdout from within python, you could try to set the capture mode to sys. My guess is that this wouldn’t be affected by stdout redirects:

from sacred.settings import SETTINGS
SETTINGS.CAPTURE_MODE = 'sys'

But I’ll mark this as a feature request nonetheless.