Interpreter doesn't exist if s3 object is not closed

I have this absolutely minimal program:

import s3fs
s3 = s3fs.S3FileSystem().open("bucket/file", "wb")
print("END")

Taking a look at this, I would expect that if you start it with python program.py, it will print an END and exit. Instead it prints END and never exits. If I put an s3.close() it works as expected.

Printing a stack trace after print gives:

Current thread 0x00007f102cfc2740 (most recent call first):
  File "/usr/local/lib/python3.10/threading.py", line 324 in wait
  File "/usr/local/lib/python3.10/threading.py", line 600 in wait
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 59 in sync
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 91 in wrapper
  File "/usr/local/lib/python3.10/site-packages/s3fs/core.py", line 1970 in commit
  File "/usr/local/lib/python3.10/site-packages/s3fs/core.py", line 1961 in _upload_chunk
  File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1522 in flush
  File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1651 in close
  File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1673 in __del__

Packages:

$ pip freeze
aiobotocore==2.0.1
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.2.0
botocore==1.22.8
charset-normalizer==2.0.9
frozenlist==1.2.0
fsspec==2021.11.1
idna==3.3
jmespath==0.10.0
multidict==5.2.0
python-dateutil==2.8.2
s3fs==2021.11.1
six==1.16.0
urllib3==1.26.7
wrapt==1.13.3
yarl==1.7.2
$ python -V
Python 3.10.1

(but it fails with older python and module versions as well, for eg. we run 2021.8.1 in prod)

[martindurant]

btw: I regularly get hangs on exit in many python multi-threaded or async applications, needing a ^C to exit.

[martindurant]

this is not in line with how other file-like objects work

Actually, it is - you stand to loose data if you exit a python program without closing files first. OK, you won’t get a hang. The trouble is, that finalizing a file might well take some time (flushing a buffer to remote), so we can’t very well just add a timeout. The best we can do is check if the event-loop/thread/client are still alive, but I’m not sure this is enough. @efiop , since you were just looking at async, you might have an idea?

Again, I’m not sure how/if we can fix this for sure - but a program trying to exit on an exception if going to be problematic in many cases.