Web Interface "Stop run" does not work

wandb --version && python --version && uname

• Weights and Biases version: 0.8.13
• Python version: 3.7.4
• Operating System: Linux

Description

Stop an in-progress run

What I Did

I ran the following demo script:

import wandb
from random import random
from time import sleep

wandb.init(project="testing-abort-job")

wandb.config.dropout = 0.2
n = 120
for i in range(1, n):
    sleep(random())
    wandb.log({"epoch": i / 10, "loss": 1/i**0.5})

I saw the project appear in the wandb web app, but when I selected “Stop run” under the context menu (using the three dots to the left of the run name) nothing happened. I did this when it was reporting step 50, and then again around step 80. It continued to run until it completed all 120 steps.

Here are the packages installed in this conda environment:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
argh                      0.26.2                   pypi_0    pypi
ca-certificates           2019.10.16                    0  
certifi                   2019.9.11                py37_0  
chardet                   3.0.4                    pypi_0    pypi
click                     7.0                      pypi_0    pypi
configparser              4.0.2                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
gitdb2                    2.0.6                    pypi_0    pypi
gitpython                 3.0.4                    pypi_0    pypi
gql                       0.1.0                    pypi_0    pypi
graphql-core              2.2.1                    pypi_0    pypi
idna                      2.8                      pypi_0    pypi
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
ncurses                   6.1                  he6710b0_1  
nvidia-ml-py3             7.352.0                  pypi_0    pypi
openssl                   1.1.1d               h7b6447c_3  
pathtools                 0.1.2                    pypi_0    pypi
pip                       19.3.1                   py37_0  
promise                   2.2.1                    pypi_0    pypi
psutil                    5.6.3            py37h7b6447c_0  
python                    3.7.4                h265db76_1  
python-dateutil           2.8.0                    pypi_0    pypi
pyyaml                    5.1.2                    pypi_0    pypi
readline                  7.0                  h7b6447c_5  
requests                  2.22.0                   pypi_0    pypi
rx                        1.6.1                    pypi_0    pypi
sentry-sdk                0.13.1                   pypi_0    pypi
setuptools                41.4.0                   py37_0  
shortuuid                 0.5.0                    pypi_0    pypi
six                       1.12.0                   pypi_0    pypi
smmap2                    2.0.5                    pypi_0    pypi
sqlite                    3.30.1               h7b6447c_0  
subprocess32              3.5.4                    pypi_0    pypi
tk                        8.6.8                hbc83047_0  
urllib3                   1.25.6                   pypi_0    pypi
wandb                     0.8.13                   pypi_0    pypi
watchdog                  0.9.0                    pypi_0    pypi
wheel                     0.33.6                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3  

[jlehrer1]

This is still a problem. When training a model tracked with Wandb on a cluster, the active state persists after the job is shut down and the web button for stop run doesn’t work.

[DanyAyala]

Giving this a bump as I’ve encountered the exact same issue when attempting to stop a hung run (> 24 hrs) using the “Stop run” button on the web interface. I don’t have physical access to the machine the experiment is running on, but I’ll speak to our internal contact about it.