S3FileSystem.checksum always returns cached value

Hi,

Encountered this unexpected behavior when using the checksum method to monitor changes in a file in a long-running app.

Description

S3FileSystem.checksum returns the same value for an object, even after the object has changed

Reproduce steps

1. Create object s3://MY_BUCKET/MY_OBJECT on s3
2. Run

import s3fs
fs=s3fs.S3FileSystem()
fs.checksum("s3://MY_BUCKET/foo")

1. Overwrite the object with new contents (not using fs, but using e.g. the aws cli)
2. In the same Python process as above, run fs.checksum("s3://MY_BUCKET/MY_OBJECT")

The results from 2 and 4 are identical, even though the file has changed.

Cause

It seems that checksum uses the default implementation of fsspec.AbstractFileSystem, which in turn creates the checksum from the output of fs.ls(detail=True). Since in the S3FileSystem implementation this gets passed a default argument refresh=False, the checksum is always computed on cached results.

Proposed solution

checksum should be implemented in S3FileSystem, in such a way that it always checks the actual state of the object, or at least provides the option to do so.

[martindurant]

I don’t think a directory has a checksum, does it? It what manner are you changing it between calls?

[martindurant]

Right, if the files change via some other process, s3fs doesn’t know about this. Please do feel free to propose changes to the documentation to clarify things.