queue: Ability to send different signals to running tasks

dvc queue kill can be used to terminate running tasks. One can also use dvc queue logs --follow to attach to the output of a running task, but CTRL+C (SIGINT) will only detach but not forward the interrupt signal to the running task (which is good). There is no option at the moment to gracefully interrupt a running task, e.g., a training stage using DVC checkpoints as one can and is being used in the user guide for DVC Experiments with checkpoints. Options for dvc queue kill which enable the user to send different signals (SIGTERM, SIGINT) to the running task similar to the Unix kill command would be useful.

Related, sending SIGINT via CTRL+C would be possible if there was a command to attach to a running task such that the signal is indeed forwarded to the task rather than handled by, e.g., dvc queue logs --follow. Possibly similar to docker attach, where one can declare --detach-keys to detach without interrupting the task. A key combination different from CTRL+C could also have been used for dvc queue logs --follow to unfollow / detach again without interruption (see also tmux) but CTRL+C would have been forwarded to the task still. This different behavior could be covered by a new dvc queue attach command (which could make dvc queue logs --follow obsolete actually).

[dberenbaum]

How important is this? It seems like a new feature request that may not be urgent.

Issues like #8612 should work even with SIGKILL. Can we hold off on this until we address the reported regressions and bugs?

[pmrowla]

Discussed this with @karajan1001, looks like there may be an issue with our SIGINT handling in dvc exp exec-run (which is used internally to run the exps from celery), so that’s where we’ll need to start for this issue. It should not be necessary for us to keep track of each child pipeline stage process ID.