Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`pytest-xdist` breaks asyncio-based code expecting to be run in the main thread

See original GitHub issue

When asyncio-based software needs to register signal handlers, calling add_signal_handler() only works when the current thread is main and raises a RuntimeError otherwise.

When I first tried integrating pytest-xdist into aiohttp’s test suite a few years ago, we’ve faced a problem that about 15% of pytest invocations would hit this and had to disable the plugin:

The reproducer is to clone aio-libs/aiohttp, add -n auto to addopts somewhere in setup.cfg, and re-run make test until you see the traceback. On aiohttp side, it’s coming from the watcher.attach_loop() call invoked from the pytest plugin @ https://github.com/aio-libs/aiohttp/blob/742a8b6/aiohttp/pytest_plugin.py#L161.

Now that I’ve had some time to try to debug what’s happening, I’ve stuck a debugger right before the line that raises RuntimeError and confirmed that the thread wasn’t main:

>>> import threading
>>> threading.enumerate()
[<_MainThread(MainThread, started 140666216200000)>, <_DummyThread(Dummy-1, started daemon 140666196751936)>]
>>> threading.current_thread()
<_DummyThread(Dummy-1, started daemon                 
140666196751936)>

@asvetlov It should be possible to fix this on the asyncio side with ThreadedChildWatcher (https://bugs.python.org/issue35621 / https://github.com/python/cpython/pull/14344) but it only appeared in the stdlib since Python 3.8: https://stackoverflow.com/a/58614689/595220.

It’s hard to pinpoint where that thread is coming from but it appears to be execnet’s fault: https://github.com/pytest-dev/pytest-xdist/blob/c5fadcd734b008251fbaa438e0806235d9bd081b/src/xdist/workermanage.py#L41 + https://github.com/pytest-dev/execnet/blob/116637ab183bc078647f281b52e662650e8178e7/execnet/multi.py#L27.

@nicoddemus do you have any insight into this? I’m rather lost at this point. Is there any way to avoid this race condition?

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:35 (29 by maintainers)

Top GitHub Comments

4reactions

graingertcommented, Jan 26, 2021

2reactions

nicoddemuscommented, Mar 31, 2021

Ahh that’s a bummer.

In my case we are not really using asyncio, but were experiencing the problem of spawning code not in the main thread.

Thanks folks for the feedback.

Top Results From Across the Web

pytest - Run tests concurrently - Stack Overflow

Definitely the simplest option is to use pytest-xdist to run tests in separate threads. You can still use async internally inside each test ......

pytest recipes — Gentoo Python Guide documentation

pytest -xdist is a plugin that makes it possible to run multiple tests in parallel. This is especially useful for programs with large...

pytest-dev - Bountysource

`pytest-xdist` breaks asyncio-based code expecting to be run in the main thread $ 0 ... when the current thread is main and raises...

pytest Documentation - Read the Docs

test run reporting finishing. Note: Calling pytest.main() will result in importing your tests and any modules that they import. Due to the.

Changelog — pytest documentation

Backward incompatible (breaking) changes will only be introduced in major versions with advance notice in the Deprecations section of releases. pytest 7.1.3 ( ......