Stopping a run from the Wandb UI doesn't work in Docker

• Weights and Biases version: 0.8.13
• Python version: 3.6.8
• Operating System: Ubuntu 16.04

Description

I would like to stop a run from the Wandb UI, where there is an option to “stop run” on the run overview page. This works for experiments started from the command line. But when I have an experiment running in a docker container, it doesn’t work.

I have started an experiment in the container with something like (kubernetes-config-style):

command: [ "/bin/bash", "-c", "--" ]
args:
 - "cd /my_code && python3 -m my_experiment_module"

When I hit the “stop run” button, I don’t see any output in the container logs - it just keeps running like nothing happened.

I know a common issue with docker containers is that you’ll send a signal to the container, but it’ll get eaten by bash and not passed on to your program. But here it doesn’t seem like that would be the issue.

[raubitsj]

Thanks for the report, we will try to reproduce using your instructions. We should be sending a signal from a spawned process back to the parent process (your experiment module) but perhaps we will need to do something slightly different in the environment you created.

[dHonerkamp]

Basically I’ve just been using wandb-docker-run -d myImage to run a few runs with wandb. But when trying to stop one of them through the web interface, the container never seemed to receive this stop signal and simply continued with the run without any indication in the logs.