S3FileSystem().exists throws Forbidden when file doesn't exists

I’m using Python’s s3fs library to check if a particular file exists in s3 with s3fs.S3FileSystem().exists(path), but I’m getting a Forbidden exception. From the stack trace, I can see it fails when calling s3’s head_object method. The documentation for head_object method says:

If the object you request does not exist, the error Amazon S3 returns depends on whether you also have the s3:ListBucket permission.

• If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 (“no such key”) error.
• If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 (“access denied”) error.

And I have the following task policy attached to my Fargate task:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListObjects",
                "s3:ListBucket",
                "s3:HeadObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

I’ve checked that the fargate task is indeed with this policy, but botocore keeps returning 403 instead of 404, even when I have the s3:ListBucket permission. Has anyone experienced this problem?

Maybe s3fs should consider 403 as the file not existing as well.

[martindurant]

To others, yes please do raise your hands if this has happened to you, on fargate or otherwise.

@ianliu “Maybe s3fs should consider 403 as the file not existing as well.”

I’m not convinced by this - the user generally needs to know the difference between something not existing and something that might exist if they had different permissions.

[martindurant]

The documentation for head_object is explicit in saying that both cases the file doesn’t exist. Can forbidden be raised in another scenario?

That is not my reading. If you see a 404, the file does not exist. If you see a 403, the file may or may not exist, but you do not have permission to know which. It is important for AWS not to leak the possible existence of files to non-privileged users.