question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Can a dedicated ThreadPoolExecutor be used in SyncToAsync?

See original GitHub issue

I think it’s common to use the @sync_to_async decorator to run Django ORM’s queries, but in the current setup that function will be ran in the global thread pool. I would like to set a dedicated thread pool just to run database queries, so I can keep the database connections persistent for some time. Django states the following about persistent connections:

Since each thread maintains its own connection, your database must support at least as many simultaneous connections as you have worker threads […] The development server creates a new thread for each request it handles, negating the effect of persistent connections. Don’t enable them during development.

In the global thread pool the threads may be busy running non query functions, so its persisted connection will not be usable. I think that could be done the same way it’s done for single_thread_executor as a class variable of SyncToAsync, so user’s could subclasss it and change the executor. Maybe allowing it to be passed throught constructor would be better.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
HMakercommented, Jan 17, 2021

I read in other issues that it is expected a process per request, so I think currently I can’t work with multi-threads in same process. Thanks for the attention.

Could you guide me to resources where I can learn more about this? What are the possible problems with multi-thread?

0reactions
HMakercommented, Jan 17, 2021

So I can’t have concurrent queries using persistent connections? I thought it would be OK because the asgiref.sync_to_async was the way, as stated in channels docs, to run DB queries in an async context. In channels 2.4.0 and asgiref 3.2.10 thread_sensitive was by default False. I don’t know django internals and I didn’t saw in any place the persistent connections were not supported in multi-threaded environments.

From what I understood thread_sensitive makes all the functions be ran in the same thread, I can’t see how it’s possible to run concurrent blocking ORM queries in an async context which is itself single thread.

In this case I am not running queries in a threaded HTTP request/response cycle, I am using django’s ORM in a non web async app that uses the global thread pool for concurrent queries.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Source code for asgiref.sync - Django documentation
In Python 3.7+, the ThreadSensitiveContext() context manager may be used to ... is async (i.e. SyncToAsync is outermost), then this will be a...
Read more >
asgiref/sync.py at main - GitHub
This context manager controls which thread pool executor is used when in ... this will be a dedicated single sub-thread that all sync...
Read more >
asgiref.sync - Django 3.1.dev documentation
This uses an asgiref # Local, not a threadlocal, so that tasks can work out what their parent used. executors = Local() def...
Read more >
Threadpool used in one @Async method getting shared with ...
I have a thread-pool initialized as a @Bean for purpose of dedicated execution of a particular @Async method
Read more >
ThreadPoolExecutor vs. AsyncIO in Python
The ThreadPoolExecutor is designed to execute ad hoc tasks, whereas async/await is a general framework for asynchronous programming. The ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found