Add a lock on tasks

I have a lot of long-running tasks that I’m running using doit.

I often accidentally launch a task while it was already running. Having two processes running the same task often lead to issues:

• use of the same resources
• second task will overwrite the results of the first task
• even if all goes well, I’ve wasted my CPU resources

I’d like to have a “locking” mechanism for a task. Is something like this already implemented ?

For now, I’m using a heuristic to check that a task isn’t already running right at the beginning of my code. But it would be nicer if the runner could do this for me, by e.g. storing a lock in the DB when a task is launched. The lock would need some kind of deadline in case doit crash and is unable to remove it, and the lock should probably be ignored with the doit run -a flag.

Is there a hook to write something before a task is run ? AFAIU value_savers are only called after the task is run.

And of course, thanks for the nice tool!

[gwenzek]

Thanks for sharing this library.

I think this can be helpful for a workaround, but the main issue is I’m not sure what to do if I see a lock file.

• I don’t want to mark the task as “uptodate” because I don’t want to trigger depending tasks
• I don’t want to return a fake “success” value because I don’t want to trigger depending tasks
• I don’t want to throw an error because it prevents doit to process other unrelated tasks

As a workaround I’m throwing an error if the lock file exists and use -c but I may end up skipping over other errors.

Currently a task has 3 possible status: {“up-to-date”, “run”, “error”} I’d like a 4th one: “delayed” which means that depending tasks can’t be executed but that doit should continue processing other tasks.

Do you think that would be a reasonable design ?

[schettino72]

I’m proposing a 4th state “delayed” which means that the task is not ready to be run. Am I clearer ?

• you a know a task is running
• you ask doit to run it again
• but you don’t want doit to execute the task

like that?i am assuming you have a complex dependency tree… otherwise you could just not run doit again. right?

not clear what would be the behavior of ‘delayed’. doit already has a delayed meaning task will be executed at a later stage by same process, seems thats not what you want.

maybe what you are looking for is what doit calls ‘ignore’… you would just need to to set it dynamicaly (not possible as of today)