FileNotFoundError when calling info

When calling info on a file that has been uploaded to S3 after info has already been called on another file in the same prefix, s3fs reports a FileNotFoundError, here’s a code snippet that reproduces the behaviour.

import s3fs
import boto3


bucket_name = "bucket"
prefix = "data"

s3 = s3fs.S3FileSystem()
bucket = boto3.resource('s3').Bucket(bucket_name)

def test(filename):
    key = f"{prefix}/{filename}"
    with open(filename, 'rb') as data:
        bucket.upload_fileobj(data, key)
    info = s3.info(f"{bucket_name}/{key}")
    print(f"{key} info: {info}")


def main():
    test("file1")
    test("file2")


if __name__ == "__main__":
    main()

output:

python s3fs_info_test.py
data4/file1 info: {'Key': 'bucket/data/file1', 'LastModified': datetime.datetime(2019, 9, 13, 17, 30, 16, tzinfo=tzutc()), 'ETag': '"664888fac652413c3a57c2e669d5d2b5"', 'Size': 151645, 'StorageClass': 'STANDARD', 'type': 'file', 'size': 151645, 'name': 'bucket/data/file1'}
Traceback (most recent call last):
  File "s3fs_info_test.py", line 25, in <module>
    main()
  File "s3fs_info_test.py", line 21, in main
    test("file2")
  File "s3fs_info_test.py", line 15, in test
    info = s3.info(f"{bucket_name}/{key}")
  File "/home/steve/.virtualenvs/partition_stats-RqvIULMs/lib/python3.7/site-packages/s3fs/core.py", line 498, in info
    return super().info(path)
  File "/home/steve/.virtualenvs/partition_stats-RqvIULMs/lib/python3.7/site-packages/fsspec/spec.py", line 509, in info
    raise FileNotFoundError(path)
FileNotFoundError: bucket/data/file2

Not sure if it’s the best way to fix it (or if it even does fix the issue) but in def info you could try wrapping return super().info(path) in a try/except and if it’s a FileNotFoundError, call info again but pass refresh=True.

The use case for us is that we have a lambda subscribed to bucket notifications for objects that are created, and what is happening is that when the lambda container is reused consecutive files that are uploaded to the same prefix fail with a FileNotFoundError. After some time goes by (long enough to get a fresh lambda container) retrying the same bucket notification event works fine.

For now we’ve pinned at 0.2.1, which doesn’t have this bug.

[martindurant]

It is a valid argument. Please make a PR with this change, and encourage other team members to discuss.

[rambej0201]

@martindurant I actually got FIleNotFoundError for some events during 05Feb2020 than I made sure about s3.invalidate_cache() on the API