vacuum does not work properly for large DBs in 0.12.x

Observed behavior

ref: https://github.com/learningequality/kolibri-server/issues/20#issuecomment-523139707 from @jredrejo :

A problem with one function that’s been executed whenever kolibri starts and keeps the db open while vacuum is done, thus vacuum does not work properly. This is only reproducible in big databases as the one you had. This problem has been fixed in kolibri ( in learningequality/kolibri#5494 ) but the fix is not included in the latest release, it will be included in the 0.13.x version.

Expected behavior

vacuum works as expected

User-facing consequences

from @intelliant01:

the bug results in all sorts of inconsistent behaviour for end users

Errors and logs

…

Steps to reproduce

Start kolibri with kolibri start --foreground Stop kolibri with kolibri stop sqlite journal files keep there and journal is not written to the database However, if kolibri is stopped using Ctrl-c in the same window where it’s started, all pending data is written to the database and journal files disappear.

Context

0.12.8, but not 0.13.x

[intelliant01]

Tested in the following environment - kolibri 0.12.9b2 deb kolibri server 0.3.2 sqlite db size ~ 2.7 GB. Hardware - Celeron Dual core + 4 GB RAM

Observations:

1. Startup time i.e. time from start to Kolibri being smoothly responsive and usable, has reduced from around 10 minutes to under 5 minutes.

kolibri.txt upon each start after the first post upgrade which involved db update and took around 9 minutes -

WARNING 2019-10-31 12:18:28,717 options Options file /home/mssadmin/.kolibri/options.ini has been updated; server restart is required before change will take effect.
INFO 2019-10-31 12:18:37,470 apps Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO 2019-10-31 12:18:39,206 cli Running 'kolibri start' as daemon (system service)
INFO 2019-10-31 12:18:39,207 cli Starting Kolibri background services
INFO 2019-10-31 12:18:39,209 cli Going to daemon mode, logging to /home/mssadmin/.kolibri/logs/kolibri.txt
INFO 2019-10-31 12:18:39,326 ping Attempting a ping.
INFO 2019-10-31 12:18:40,176 utils Started new JOBCHECKER thread ID#6ae8975f4569472e9236c2e7b0d994d2
INFO 2019-10-31 12:18:40,178 server Starting Kolibri 0.12.9b2
INFO 2019-10-31 12:19:02,129 apps Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO 2019-10-31 12:22:31,282 vacuumsqlite Sqlite database Vacuum finished.
INFO 2019-10-31 12:22:31,408 vacuumsqlite Next Vacuum at 3AM local server time (in 877.4666666666667 minutes).
INFO 2019-10-31 12:22:47,148 ping Ping succeeded! (response: {'id': 427166, 'messages': []})
INFO 2019-10-31 12:23:09,661 ping Sleeping for 1440.0 minutes.

1. No more db locking error in daemon.txt

INFO     Attempting a ping.
INFO     Starting Kolibri 0.12.9b2
INFO     Started new JOBCHECKER thread ID#6ae8975f4569472e9236c2e7b0d994d2
INFO     Sqlite database Vacuum finished.
INFO     Next Vacuum at 3AM local server time (in 877.4666666666667 minutes).
INFO     Ping succeeded! (response: {'id': 427166, 'messages': []})
INFO     Sleeping for 1440.0 minutes.

1. Shutdown is clean except the timeout for kolibri-server despite increasing TimeoutStopSec=60 in kolibri-server.service

kolibri.txt

INFO 2019-10-31 12:25:33,691 apps Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO 2019-10-31 12:25:36,845 cli Kolibri background services have successfully been stopped.
INFO 2019-10-31 12:25:45,208 apps Running Kolibri with the following settings: kolibri.deployment.default.settings.base

# systemctl status kolibri-server kolibri
● kolibri-server.service - A high performance web server setup for Kolibri
   Loaded: loaded (/lib/systemd/system/kolibri-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Thu 2019-10-31 12:26:07 IST; 5min ago
  Process: 7342 ExecStop=/etc/init.d/kolibri-server stop (code=exited, status=0/SUCCESS)
  Process: 7052 ExecStart=/etc/init.d/kolibri-server start (code=exited, status=0/SUCCESS)
 Main PID: 7198 (code=killed, signal=KILL)

Oct 31 12:18:26 server su[7073]: + ??? root:mssadmin
Oct 31 12:18:26 server su[7073]: pam_unix(su:session): session opened for user mssadmin by (uid=0)
Oct 31 12:18:28 server kolibri-server[7052]: WARNING  Options file /home/mssadmin/.kolibri/options.ini has been updated; server restart is required before change will take effe
Oct 31 12:18:29 server kolibri-server[7052]: debconf: DbDriver "passwords" warning: could not open /var/cache/debconf/passwords.dat: Permission denied
Oct 31 12:18:56 server kolibri-server[7052]: [uWSGI] getting INI configuration from /etc/kolibri/dist/uwsgi.ini
Oct 31 12:18:56 server systemd[1]: Started A high performance web server setup for Kolibri.
Oct 31 12:25:21 server systemd[1]: Stopping A high performance web server setup for Kolibri...
Oct 31 12:26:07 server systemd[1]: kolibri-server.service: Main process exited, code=killed, status=9/KILL
Oct 31 12:26:07 server systemd[1]: kolibri-server.service: Failed with result 'signal'.
Oct 31 12:26:07 server systemd[1]: Stopped A high performance web server setup for Kolibri.

● kolibri.service - LSB: kolibri daemon, an offline education platform
   Loaded: loaded (/etc/init.d/kolibri; disabled; vendor preset: enabled)
   Active: inactive (dead)

Oct 31 12:18:39 server kolibri[7124]: INFO     Running 'kolibri start' as daemon (system service)
Oct 31 12:18:39 server kolibri[7124]: INFO     Starting Kolibri background services
Oct 31 12:18:39 server kolibri[7124]: INFO     Going to daemon mode, logging to /home/mssadmin/.kolibri/logs/kolibri.txt
Oct 31 12:18:39 server systemd[1]: Started LSB: kolibri daemon, an offline education platform.
Oct 31 12:25:27 server systemd[1]: Stopping LSB: kolibri daemon, an offline education platform...
Oct 31 12:25:27 server su[7416]: Successful su for mssadmin by root
Oct 31 12:25:27 server su[7416]: + ??? root:mssadmin
Oct 31 12:25:33 server kolibri[7385]: INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
Oct 31 12:25:36 server kolibri[7385]: INFO     Kolibri background services have successfully been stopped.
Oct 31 12:25:37 server systemd[1]: Stopped LSB: kolibri daemon, an offline education platform.

1. A line in kolibri.txt is INFO 2019-10-30 21:21:17,081 cli Going to daemon mode, logging to /home/mssadmin/.kolibri/logs/kolibri.txt . I think it should be daemon.txt instead of kolibri.txt

2. If possible, addition of timestamps in daemon.txt shall help to co-orelate with the events being logged in kolibri.txt.

[indirectlylit]

We want to backport the fixes because 0.13.0 won’t be released for a few months. (we don’t have an official concept of “LTS” with regard to Kolibri releases)