Extremely slow line iteration on Windows/ProactorEventLoop
See original GitHub issueWindows 7 64-bit, Python 3.7.0 64-bit, aiofiles 0.4.0
Test program:
import asyncio, aiofiles, timeit
with open('sample.txt', 'w') as f:
for i in range(10000):
print('A'*100, file=f)
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
def traditional_read():
with open('sample.txt', encoding='utf-8') as f:
for ln in f:
pass
async def aiofiles_read():
async with aiofiles.open('sample.txt', encoding='utf=8') as f:
async for ln in f:
pass
print('traditional:', timeit.timeit('traditional_read()', number=1, globals=globals()))
print('aiofiles:', timeit.timeit('loop.run_until_complete(aiofiles_read())', number=1, globals=globals()))
Output:
traditional: 0.005477579999999982
aiofiles: 1.563328274
The sample file is not a very big one (less than 1MB), but it contains a bit “too many” lines. So is the result caused by excess thread context switches? Can this be avoided by some approach?
Issue Analytics
- State:
- Created 5 years ago
- Reactions:3
- Comments:6
Top Results From Across the Web
PyPy-3.8-7.3.7 : Problem with asyncio.ProactorEventLoop on ...
This is the default event loop on Windows in Python 3.8. By manually setting the event loop to asyncio.SelectorEventLoop (using asyncio.
Read more >ProactorEventLoop raises unhandled ConnectionResetError
Even simpler, the following code will crash after so many iterations: ``` import asyncio loop = asyncio.get_event_loop() while True: loop.
Read more >aiofiles - Bountysource
One of the potential uses of aiofiles is to read sys.stdin asynchronously, which is currently not ... However, I find that on windows,...
Read more >Slow for loop iteration in batch file - powershell - Stack Overflow
Since you say you are new to this. It's really important you get some ramp up on the topic. Using either all the...
Read more >Changelog — aiohttp 3.8.3 documentation
Note that Pure-Python mode is significantly slower than compiled one. ... iterate over the content with readany instead of readline to avoid Line...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Thanks @MrMrRobat for investigating.
The reason for this issue is that for each line
aiofiles
will start new task in executor, which is a huge overhead just for one line reading:Code above is just for demonstration, original code is: https://github.com/Tinche/aiofiles/blob/d85a65f1c7c84a98c979e19c12fe1d2f642b9cec/aiofiles/threadpool/text.py#L6-L7 https://github.com/Tinche/aiofiles/blob/145107595fe72b877005fb4ebf9d77ffb754689a/aiofiles/threadpool/utils.py#L5-L10 https://github.com/Tinche/aiofiles/blob/145107595fe72b877005fb4ebf9d77ffb754689a/aiofiles/threadpool/utils.py#L31-L37