Performance bottleneck with mono threaded Downloader Middleware

Summary

There seems to be a performance bottleneck in the case where a Downloader Middleware is I/O heavy: I have a custom cache storage (very similar to FilesystemCacheStorage) which does some I/O. In the cases where the cache hit ratio is high, this cache becomes the overwhelming bottleneck.

Additional context

I tried playing with both REACTOR_THREADPOOL_MAXSIZE and CONCURRENT_REQUESTS but it seems that no multi-threading is taking place to invoke the cache middleware since it’s only ran from the main thread. I am not very familiar with Reactor thread pooling but I cannot find a nob to turn on multithreading at that level.

[wRAR]

You could always write async code in middlewares, using Deferreds.

There is no such thing as “reactor threads”. Native Python coroutines get wrapped into Twisted Deferreds.

i get non blocking I/O

Not necessarily. That’s why I wrote “Assuming your I/O-bound code is synchronous” and “asyncio doesn’t make your code asynchronous by itself”.

and muti threading in the middleware for free

You already have callInThread/deferToThread.

[yonromai]

Perfect, thanks so much for your help @wRAR