test_worker_heartbeat_after_cancel hangs on MacOS 3.7/3.8

With the latest CI improvements, I receive failures which look like this. I assume this is a GH actions script this is pointing to but I have no idea what this particular error is about.

@crusaderky have you encountered anything like this in your tests?

https://github.com/fjetter/distributed/runs/3064775791?check_suite_focus=true

/Users/runner/work/_temp/eec694e8-9cbc-41e2-9505-f187841b5aa3.sh: line 9:  6519 Abort trap: 6           pytest distributed -m "not avoid_ci" --runslow --junitxml reports/pytest.xml -o junit_suite_name=$TEST_ID
distributed/tests/test_scheduler.py::test_worker_heartbeat_after_cancel 
Error: Process completed with exit code 134.

[crusaderky]

That’s pytest-timeout that just sent SIGABRT. I was hoping it would kill off the individual test on MacOS like it does on Linux. Clearly it’s not like that. I will revert to timeout_method = thread for MacOS.

[fjetter]

I couldn’t reproduce the hanging test locally. However, what’s interesting, is that I could see a lot of “loop unresponsive” messages when running the test under a lot of load. Still, idk why it hung. The new test logic is less prone to instabilities