Hitting OSError: [Errno 24] Too many open files

Places this has surfaced:

• @catherinewu can reproduce reliably by running pytest python_modules/libraries/dagster-dask/dagster_dask_tests/test_execute.py (but I cannot)
• @helloworld ran into it while clicking around dagit
• @yuhan ran into it while running dagster core tests
• @alangenfeld ran into it with graphql tests

It’s not clear that these are all related.

Here’s one way to reproduce:

ulimit 256
pytest -s python_modules/dagster-graphql/dagster_graphql_tests/graphql/

To observe file descriptors leaking, run this in one terminal:

pytest -s python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_config_types.py

and repeatedly run this in another terminal:

lsof -p $(ps -A | grep pytest | grep -v grep | awk '{print $1}') > $(date +"%s")

Observe that the number of open file descriptors given by lsof increases throughout the test, with about one new temporary directory sticking around per test.

[sryza]

Found it: https://dagster.phacility.com/D3472

[schrockn]

https://dagster.phacility.com/D3472