Tasks don't complete in Device - Channels module

Observed behavior

Tasks are no longer completing after an abrupt shutdown (power to the server unplugged) while an update task was ongoing. The tasks were completing and had even cleared them prior to the abrupt shutdown.

Expected behavior

Tasks should complete.

User-facing consequences

Can’t accomplish any task in the Device -> Channels module.

Errors and logs

Herein are attached the logs of the last few days ever since the problem started. kolibri-2020-01-25.txt kolibri-2020-01-26.txt kolibri-2020-01-27.txt kolibri-2020-01-23.txt kolibri-2020-01-24.txt

Steps to reproduce

1. Start a channel content update task
2. Simulate an abrupt shutdown of Kolibri while the task is still underway (Unplug the device)
3. Start device and Kolibri
4. No new tasks shall complete thereafter

Context

Kolibri server - 0.3.6 Kolibri - 0.13.0 Ubuntu 18.04.3

[jredrejo]

After some live debugging with the @intelliant01 installation, the problem was on this line https://github.com/learningequality/kolibri/blob/release-v0.13.x/kolibri/core/content/management/commands/importchannel.py#L179

importchannel tries to acquire a lock to do the importation and could not do it because the database said a lock was acquired by another process with a different pid. Diskcache does not return any error in this case, it just keeps the acquire routine in an infinite loop: https://github.com/grantjenks/python-diskcache/blob/master/diskcache/recipes.py#L145-L154 until the lock is released.

The problem appears because when the abrupt shutdown happened, kolibri was interrupted while doing the import and had a pid that was inserted in the cache.db to lock the process. After kolibri restarted the pid was not the same and we were in a deadlock.

I think the solution should be deleting cache.db whenever kolibri starts, to avoid this kind of problems, but I’d like to hear more opinions, specially from @rtibbles who implemented the locking here.

btw, this bug may happen with all the tasks using db_task_write_lock, not only importing channels, also deleting them, trying to vacuum, importing content, etc.

I am attaching here the bad db to help debugging

[intelliant01]

My experience is similar to point 1 therein, though I have not tried the uninstall and reinstall step -

When I import several resources at the same time the Task Manager gets stuck at 100%. Cancelling the import doesn’t work. Only after restarting the computer does the task disappear. But afterwards I get the same problem with every task. So completely unable to import anything. I have to uninstall (deleting all imports) and reinstall Kolibri to be able to import again.

However I do remember being able to start and complete multiple imports. They did execute sequentially but they did get enlisted in the tasks list simultaneously and did complete. It was only after a hard shutdown that I experienced the problem.

Shall try to provide a gif soon as I may be away from the problem system for the next two days.