KeyError: 'ETag'

What happened: I get a KeyError when trying to copy files from bucket to local:

>       _, _, parts_suffix = info["ETag"].strip('"').partition("-")
E       KeyError: 'ETag'

What you expected to happen: files get copied

Minimal Complete Verifiable Example:

from s3fs import S3FileSystem
s3 = S3FileSystem()
s3.copy(input_path, '/tmp/foobar', recursive=True)

Anything else we need to know?: in the _info() function, when the ls was cached, the response never includes the ‘Etag’ key, so I think it’s a bug to rely on it always being there.

Environment:

• s3fs version: 2021.4.0
• Python version: 3.6.10
• Operating System: Mac
• Install method (conda, pip, source): pip

[martindurant]

It does seem reasonable not to depend on the existence of ETag, if some implementations do not have it. I imagine that for moto, this is version-dependent, since we are not seeing the problem in CI.

From the description, I couldn’t tell if the problem was with attempting to copy a folder (which is not an S3 thing) or one of the constituent files.

[machielg]

in the _info() function, when the ls was cached, the response never includes the ‘Etag’ key, so I think it’s a bug to rely on it always being there.

I think this is the actual issue. @machielg can you share the fs.info(input_path) too? I guess as an end result we could just do info.get("ETag", "") though I wonder why the etag is missing in the first place.

s3.info(input_path)
{'Key': 'source_bucket/processed/scoring_set/partitioning_date=2021-04-18', 'name': 'source_bucket/processed/scoring_set/partitioning_date=2021-04-18', 'type': 'directory', 'Size': 0, 'size': 0, 'StorageClass': 'DIRECTORY'}