Can a dedicated ThreadPoolExecutor be used in SyncToAsync?
See original GitHub issueI think it’s common to use the @sync_to_async
decorator to run Django ORM’s queries, but in the current setup that function will be ran in the global thread pool. I would like to set a dedicated thread pool just to run database queries, so I can keep the database connections persistent for some time. Django states the following about persistent connections:
Since each thread maintains its own connection, your database must support at least as many simultaneous connections as you have worker threads […] The development server creates a new thread for each request it handles, negating the effect of persistent connections. Don’t enable them during development.
In the global thread pool the threads may be busy running non query functions, so its persisted connection will not be usable.
I think that could be done the same way it’s done for single_thread_executor
as a class variable of SyncToAsync
, so user’s could subclasss it and change the executor. Maybe allowing it to be passed throught constructor would be better.
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (2 by maintainers)
I read in other issues that it is expected a process per request, so I think currently I can’t work with multi-threads in same process. Thanks for the attention.
Could you guide me to resources where I can learn more about this? What are the possible problems with multi-thread?
So I can’t have concurrent queries using persistent connections? I thought it would be OK because the
asgiref.sync_to_async
was the way, as stated in channels docs, to run DB queries in an async context. In channels 2.4.0 and asgiref 3.2.10thread_sensitive
was by default False. I don’t know django internals and I didn’t saw in any place the persistent connections were not supported in multi-threaded environments.From what I understood
thread_sensitive
makes all the functions be ran in the same thread, I can’t see how it’s possible to run concurrent blocking ORM queries in an async context which is itself single thread.In this case I am not running queries in a threaded HTTP request/response cycle, I am using django’s ORM in a non web async app that uses the global thread pool for concurrent queries.