Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use thread-local connections for SQLite backend

See original GitHub issue

I believe multi-threaded performance with SQLite could be significantly improved by using thread-local connections, and (optionally?) keep them open until explicitly closed. Currently, a new connection is created for every read/write operation (unless bulk_commit() is used).

Another possible improvement to look into (maybe as a separate issue) would be to add a write queue shared across threads instead of using a lock to prevent concurrent writes.

Issue Analytics

State:
Created 2 years ago
Comments:5 (2 by maintainers)

Top GitHub Comments

1reaction

JWCookcommented, Apr 21, 2021

I ran some benchmarks to compare check_same_thread=False with a lock vs. defaults and no lock. Even with 20 threads running at once, averaged over 160 writes, there’s almost no performance difference; about 10 microseconds per write, which could just be due to randomness.

I also timed bulk_commit() vs. committing after each write, and you’re right, bulk_commit() is still many times faster.

1reaction

jsemriccommented, Apr 20, 2021

I’d like to work on this issue

Top Results From Across the Web

Multi-threaded Access to an SQLite3 Database

Thread-local connections are very portable: the only requirement is that SQLite has been built with threading support enabled. Connections are ...

Using SQLite In Multi-Threaded Applications

In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or...

SQLite - requests-cache 0.9.7 documentation

SQLite supports concurrent access, so it is safe to use from a multi-threaded and/or multi-process application. It supports unlimited concurrent reads. Writes, ...

ThreadLocal + java.sql.Connection + servlet filter = 2009?

I will have to check it out, I'm trying to avoid large-ish frameworks in this app (thus going with plain servlets) but if...

Additional Persistence Techniques — SQLAlchemy 2.0 ...

SQLite has limitations in combining the use of RETURNING with triggers, such that the RETURNING clause will not have the INSERTed value ...