Bugfix: Retry on "database locked" error (or add support for PostgreSQL/MySQL DB backend)

when using the documented method to add or update links in parallel , or using the gui while updating on the cli , plenty of database locked errors , and since the somehow lengthy ( min. 500ms per page only to check , up to 6*120 s with timeout 2m ) a simple RETRY_DB_WRITE variable and the corresponding code would help

(or just switch to SQL , since it is way faster , especially when running 20Mbyte+ sqlite files from SD/MMC )

[pirate]

Note I’ve added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.

[pirate]

While Postgres is better at concurrent writes (not necessarily reads), it’s definitely not a clearcut winner on “better at maintainability and security”. Postgres is significantly less tested than SQLite. I think you forget that Airbus planes, rockets, boats, most iOS apps and other native apps, and tons of modern critical infrastructure run on SQLite!

• https://www.sqlite.org/testing.html
• https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-qps-on-a-single-server/
• https://antonz.org/sqlite-is-not-a-toy-database/
• https://github.com/rqlite/rqlite

We might add Postgres/MySQL support in the future, but it’s not a priority because ArchiveBox is focused on long-term data format durability above all else, and SQLite is the clear winner in that regard, having been standardized to the point of wide adoption far beyond Postgres and MySQL and others. Side note, I love Postgres, but honestly MySQL is slightly better for the ArchiveBox use-case, InnoDB compression is better, and it can handle many more concurrent threads than Postgres with lower resource usage on smaller machines/containers.

I’m hesitant to link you to the docs on changing the database unless you’re already experienced at doing that with Django and can handle any bugs that come up on your own, because I’m not ready to provide support for people running other databases, it’s just to much extra work for a solo-maintainer/free-time project.