Autovacuum not reclaiming disk space

This is a…

[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Documentation issue or request

Description

We had a Syndesis instance (1.4.10, 76bc7c6f389f65ba0518bdd05823d247f4870a90) running for ~3 days (2 days 17 hours) with 6 integrations generating 730K messages.

The syndesis-db (1GiB) volume was on the verge of running out of disk space (12MiB left). And VACUUM FULL jsondb could not be run:

ERROR:  could not extend file "base/16385/16845": No space left on device
HINT:  Check free disk space.
STATEMENT:  VACUUM FULL ANALYSE jsondb;

Fetching activities resulted in 504 (Gateway timeout).

With ~30K rows in jsondb table the overwhelming majority of disk space was taken up with deleted rows. That were not vacuumed by autovacuum.

The only way to get the stream of activities was to copy the table content to a new table, drop the jsondb table, rename the new table to jsondb and recreate indexes.

i.e.

> SELECT * INTO jsondb2 FROM jsondb;
> DROP TABLE jsondb;
> ALTER TABLE jsondb2 RENAME TO jsondb;
> CREATE UNIQUE INDEX jsondb_pkey ON jsondb USING btree (path);
> CREATE INDEX jsondb_idx ON jsondb USING btree (idx, value) WHERE (idx IS NOT NULL)

I think we should try to tune the autovacuum so that the disk space is reclaimed (autovacuum will never perform VACUUM FULL) or bring back the DB maintenance job to perform VACUUM FULL.

The level of severity depends on how we envision the users using Syndesis, if this level of activity is acceptable (looks to me like it should be), then this feels like p0.

[KurtStam]

+1 to won’t fix. But let’s open a new issue that references this one that talks about alternative logging using maybe elastic. We’ve been abusing the DB long enough for this.

Would a quick intermediate step be to use a different table for logging (not the jsondb), and explicitly set no locking on the insert ?

[zregvart]

+1 to won’t fix. But let’s open a new issue that references this one that talks about alternative logging using maybe elastic. We’ve been abusing the DB long enough for this.

I’ve created #4011

Would a quick intermediate step be to use a different table for logging (not the jsondb), and explicitly set no locking on the insert ?

Nope, that table would grow much the same as the current one does, and row level locks are explicit.