Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ctrl+C does not cancel workflow model training.

See original GitHub issue

Describe the bug When executing a python script that uses a MONAI workflow, Ctrl+C in terminal does not cancel the script. Instead, the current epoch is skipped and training resumes at the next epoch.

To Reproduce Steps to reproduce the behavior:

  1. Execute MONAI/examples/workflows/unet_training_dict.py in the foreground.
  2. After training starts, issue SIGINT or ctrl+c to foreground process.
  3. See current epoch cancel, and training jump to a future epoch.

Inconsistent - sometimes training stops, sometimes training jumps to future epoch.

Expected behavior After pressing ctrl+c, script executes end training handlers and quits out.

Screenshots

Log output. Ctrl+C shown as ^C

-> % python unet_training_dict.py
MONAI version: 0.2.0+87.g97dffd0.dirty
Python version: 3.8.3 (default, May 19 2020, 18:47:26)  [GCC 7.3.0]
Numpy version: 1.18.5
Pytorch version: 1.5.0

Optional dependencies:
Pytorch Ignite version: 0.3.0
Nibabel version: 3.1.0
scikit-image version: 0.17.2
Pillow version: 7.1.2
Tensorboard version: 2.2.2

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

generating synthetic data to /tmp/tmpvmr5kwuk (this may take a while)
10/10 Load and cache transformed data:  [==============================]
20/20 Load and cache transformed data:  [==============================]
INFO:ignite.engine.engine.SupervisedTrainer:Engine run resuming from iteration 0, epoch 0 until 5 epochs
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 1/5, Iter: 1/10 -- train_loss: 0.6313
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 1/5, Iter: 2/10 -- train_loss: 0.6203
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 1/5, Iter: 3/10 -- train_loss: 0.5297
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 1/5, Iter: 4/10 -- train_loss: 0.6230
^CERROR:ignite.engine.engine.SupervisedTrainer:Current run is terminating due to exception: .
ERROR:ignite.engine.engine.SupervisedTrainer:Exception:
Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 689, in _run_once_on_dataset
    self._fire_event(Events.ITERATION_COMPLETED)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 607, in _fire_event
    func(self, *(event_args + args), **kwargs)
  File "/home/gagan/code/MONAI/monai/engines/workflow.py", line 120, in run_post_transform
    engine.state.output = apply_transform(post_transform, engine.state.output)
  File "/home/gagan/code/MONAI/monai/transforms/utils.py", line 289, in apply_transform
    return transform(data)
  File "/home/gagan/code/MONAI/monai/transforms/compose.py", line 232, in __call__
    input_ = apply_transform(_transform, input_)
  File "/home/gagan/code/MONAI/monai/transforms/utils.py", line 289, in apply_transform
    return transform(data)
  File "/home/gagan/code/MONAI/monai/transforms/post/dictionary.py", line 205, in __call__
    d[key] = self.converter(d[key])
  File "/home/gagan/code/MONAI/monai/transforms/post/array.py", line 289, in __call__
    mask = get_largest_connected_component_mask(foreground, self.connectivity)
  File "/home/gagan/code/MONAI/monai/transforms/utils.py", line 479, in get_largest_connected_component_mask
    item = measure.label(item, connectivity=connectivity)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/skimage/measure/_label.py", line 93, in label
    return clabel(input, neighbors, background, return_num, connectivity)
KeyboardInterrupt
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[1] Complete. Time taken: 00:00:04
INFO:ignite.engine.engine.SupervisedTrainer:Got new best metric of train_acc: 0.6335590857046621
INFO:ignite.engine.engine.SupervisedTrainer:Current learning rate: 0.001
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[1] Metrics -- train_acc: 0.6336
INFO:ignite.engine.engine.SupervisedTrainer:Key metric: train_acc best value: 0.6335590857046621 at epoch: 1
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 2/5, Iter: 6/10 -- train_loss: 0.6213
ERROR:ignite.engine.engine.SupervisedTrainer:Current run is terminating due to exception: DataLoader worker (pid(s) 213446, 213447, 213448, 213449) exited unexpectedly.
ERROR:ignite.engine.engine.SupervisedTrainer:Exception: DataLoader worker (pid(s) 213446, 213447, 213448, 213449) exited unexpectedly
Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
    fd = df.detach()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 655, in _run_once_on_dataset
    batch = next(self._dataloader_iter)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 213446, 213447, 213448, 213449) exited unexpectedly
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[2] Complete. Time taken: 00:00:00
INFO:ignite.engine.engine.SupervisedTrainer:Got new best metric of train_acc: 0.654775548864294
INFO:ignite.engine.engine.SupervisedTrainer:Current learning rate: 0.0001
INFO:ignite.engine.engine.SupervisedEvaluator:Engine run resuming from iteration 0, epoch 1 until 2 epochs
INFO:ignite.engine.engine.SupervisedEvaluator:Epoch[2] Complete. Time taken: 00:00:02
INFO:ignite.engine.engine.SupervisedEvaluator:Got new best metric of val_mean_dice: 0.3548033036291599
INFO:ignite.engine.engine.SupervisedEvaluator:Epoch[2] Metrics -- val_acc: 0.6466 val_mean_dice: 0.3548
INFO:ignite.engine.engine.SupervisedEvaluator:Key metric: val_mean_dice best value: 0.3548033036291599 at epoch: 2
INFO:ignite.engine.engine.SupervisedEvaluator:Engine run complete. Time taken 00:00:02
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[2] Metrics -- train_acc: 0.6548
INFO:ignite.engine.engine.SupervisedTrainer:Key metric: train_acc best value: 0.654775548864294 at epoch: 2
INFO:ignite.engine.engine.SupervisedTrainer:Saved checkpoint at epoch: 2
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 3/5, Iter: 7/10 -- train_loss: 0.5775
INFO:ignite.engine.engine.SupervisedTrainer:Epoch: 3/5, Iter: 8/10 -- train_loss: 0.5359
^CERROR:ignite.engine.engine.SupervisedTrainer:Current run is terminating due to exception: .
ERROR:ignite.engine.engine.SupervisedTrainer:Exception:
Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 688, in _run_once_on_dataset
    self.state.output = self._process_function(self, self.state.batch)
  File "/home/gagan/code/MONAI/monai/engines/trainer.py", line 154, in _iteration
    return {Keys.IMAGE: inputs, Keys.LABEL: targets, Keys.PRED: predictions, Keys.LOSS: loss.item()}
KeyboardInterrupt
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[3] Complete. Time taken: 00:00:02
INFO:ignite.engine.engine.SupervisedTrainer:Got new best metric of train_acc: 0.6672782897949219
INFO:ignite.engine.engine.SupervisedTrainer:Current learning rate: 0.0001
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[3] Metrics -- train_acc: 0.6673
INFO:ignite.engine.engine.SupervisedTrainer:Key metric: train_acc best value: 0.6672782897949219 at epoch: 3
ERROR:ignite.engine.engine.SupervisedTrainer:Current run is terminating due to exception: DataLoader worker (pid(s) 213534, 213535, 213536) exited unexpectedly.
ERROR:ignite.engine.engine.SupervisedTrainer:Exception: DataLoader worker (pid(s) 213534, 213535, 213536) exited unexpectedly
Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
    fd = df.detach()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 655, in _run_once_on_dataset
    batch = next(self._dataloader_iter)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 213534, 213535, 213536) exited unexpectedly
INFO:ignite.engine.engine.SupervisedTrainer:Epoch[4] Complete. Time taken: 00:00:00
ERROR:ignite.engine.engine.SupervisedTrainer:Engine run is terminating due to exception: Accuracy must have at least one example before it can be computed..
ERROR:ignite.engine.engine.SupervisedTrainer:Exception: Accuracy must have at least one example before it can be computed.
Traceback (most recent call last):
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 942, in _internal_run
    self._fire_event(Events.EPOCH_COMPLETED)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/engine/engine.py", line 607, in _fire_event
    func(self, *(event_args + args), **kwargs)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/metrics/metric.py", line 128, in completed
    result = self.compute()
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/metrics/metric.py", line 229, in another_wrapper
    return func(self, *args, **kwargs)
  File "/home/gagan/anaconda3/envs/monai/lib/python3.8/site-packages/ignite/metrics/accuracy.py", line 157, in compute
    raise NotComputableError('Accuracy must have at least one example before it can be computed.')
ignite.exceptions.NotComputableError: Accuracy must have at least one example before it can be computed.

Environment (please complete the following information):

  • OS: Debian buster
  • Python version: 3.8.3
  • MONAI version: master branch for the last few weeks.
  • CUDA/cuDNN version: 11.0
  • GPU models and configuration: Slot0 GTX 1080TI and Slot1 GTX 1070.

Additional context I noticed problem first when developing GanTrainer. I spam Ctrl+C to cancel training operation.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
vfdev-5commented, Aug 11, 2020

@Nic-Ma the behaviour looks strange, let me run it from my side and inspect what happens. Normally, 1 or 2 Ctrl+C is sufficient to stop the training.

1reaction
Nic-Macommented, Aug 12, 2020

Hi @vfdev-5 ,

I raised an issue to ignite: https://github.com/pytorch/ignite/issues/1234. Thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

can't use Ctrl+C to stop training process · Issue #5615 - GitHub
But when I hit Ctrl+C in the training process, it would not exit from the program and would stuck at saving caffemodel and...
Read more >
What to do when Ctrl-C won't kill running job? - Ask Ubuntu
That way, when you try to kill a process, the CTRL + C doesn't work. Try to replace the copy shortcut to CTRL...
Read more >
Photoshop Keyboard Shortcuts 2023 (Updated)
These Photoshop keyboard shortcuts are used primarily for moving objects or ... If the Move tool is not active, hold Command (macOS) or...
Read more >
Important SAP Keyboard Shortcuts List - STechies
This tutorial contains an Important SAP KEYBOARD SHORTCUTS List or SAP Buttons. A keyboard shortcut is a key or combination of keys that...
Read more >
AutoCAD Keyboard Commands & Shortcuts Guide | Autodesk
Manage Workflow ; Ctrl+Shift+V · Paste data as block ; Ctrl+Z · Undo last action ; Ctrl+Y · Redo last action ; Ctrl+[,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Move jupyter notebooks out of the project-monai/monai repo

Next page

TypeError in transforms notebook