Extremely slow line iteration on Windows/ProactorEventLoop

Windows 7 64-bit, Python 3.7.0 64-bit, aiofiles 0.4.0

Test program:

import asyncio, aiofiles, timeit

with open('sample.txt', 'w') as f:
	for i in range(10000):
		print('A'*100, file=f)

loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)

def traditional_read():
	with open('sample.txt', encoding='utf-8') as f:
		for ln in f:
			pass

async def aiofiles_read():
	async with aiofiles.open('sample.txt', encoding='utf=8') as f:
		async for ln in f:
			pass

print('traditional:', timeit.timeit('traditional_read()', number=1, globals=globals()))
print('aiofiles:', timeit.timeit('loop.run_until_complete(aiofiles_read())', number=1, globals=globals()))

Output:

traditional: 0.005477579999999982
aiofiles: 1.563328274

The sample file is not a very big one (less than 1MB), but it contains a bit “too many” lines. So is the result caused by excess thread context switches? Can this be avoided by some approach?

[espdev]

Thanks @MrMrRobat for investigating.

[Bobronium]

The reason for this issue is that for each line aiofiles will start new task in executor, which is a huge overhead just for one line reading:

loop.run_in_executor(executor, file.readline)

Code above is just for demonstration, original code is: https://github.com/Tinche/aiofiles/blob/d85a65f1c7c84a98c979e19c12fe1d2f642b9cec/aiofiles/threadpool/text.py#L6-L7 https://github.com/Tinche/aiofiles/blob/145107595fe72b877005fb4ebf9d77ffb754689a/aiofiles/threadpool/utils.py#L5-L10 https://github.com/Tinche/aiofiles/blob/145107595fe72b877005fb4ebf9d77ffb754689a/aiofiles/threadpool/utils.py#L31-L37