Writing large file stuck in retry

>>> p = S3Path("/foo/bar")

This works

>>> with p.open('w') as fp:
...   json.dump([None]*10,fp)


This get stuck for a long time

>>> with p.open('w') as fp:
...   json.dump([None]*5000,fp)


This works well also

>>> p.write_text(json.dumps([None]*5000))

When it is stuck, I’ve seen a lot of log messages like this.

DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:botocore.awsrequest:Waiting for 100 Continue response.
DEBUG:botocore.awsrequest:100 Continue response seen, now sending request body.

This looks like an infinite loop. But after a long time, it completed successfully.

[impredicative]

For writing small files, s3path seems fine as is. For a single-step write, boto3 works. For a streaming write smart_open is best.

In all cases I rely on s3path to manipulate paths and generate URIs.

[markopy]

I think this is the same bug I ran into. I noticed it continuously uploading data and never finishing.

Using smart_open might be a good idea since it is much more widely used and tested. It does magic things though like decompression of certain file types based on extension which would need to be disabled.