Disallowing ListObjectsV2 at the root of the bucket makes s3fs attempt to create a bucket

What happened:

It’s not uncommon to have a bucket that disallows listing the root, but allows listing a specific prefix. In this case s3fs will fail any writes and will attempt to create the bucket, which often fails with a completely different error.

What you expected to happen:

Falling back to creating a bucket is very strange behaviour. I imagine it’s legacy and impossible to change, but I would expect that s3fs does not require full list objects permissions over the bucket to perform any writes.

Minimal Complete Verifiable Example:

In [1]: import s3fs

In [2]: s3 = s3fs.S3FileSystem(anon=False)

In [5]: s3.mkdirs("s3://s3fs-test-bucket-123/foo/bar")
2021-09-17 17:34:50,222 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - () - {'MaxKeys': 1, 'Bucket': 's3fs-test-bucket-123'}
2021-09-17 17:34:50,516 - s3fs - DEBUG - _call_s3 -- Nonretryable error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
2021-09-17 17:34:50,516 - s3fs - DEBUG - _call_s3 -- CALL: create_bucket - () - {'Bucket': 's3fs-test-bucket-123', 'ACL': ''}
2021-09-17 17:34:50,576 - s3fs - DEBUG - _call_s3 -- Nonretryable error: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

The full traceback is like so:

File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/dask/dataframe/io/parquet/arrow.py", line 819, in initialize_write
    fs.mkdirs(path, exist_ok=True)
  File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/fsspec/spec.py", line 1159, in mkdirs
    return self.makedirs(path, exist_ok=exist_ok)
  File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/fsspec/asyn.py", line 88, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/fsspec/asyn.py", line 69, in sync
    raise result[0]
  File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/s3fs/core.py", line 731, in _makedirs
    await self._mkdir(path, create_parents=True)
  File "/home/app/.cache/pypoetry/virtualenvs/x/lib/python3.9/site-packages/s3fs/core.py", line 716, in _mkdir
    await self._call_s3("create_bucket", **params)

It seems like it’s failing to detect the bucket exists on this line. There are much better methods to detect if a bucket exists, like get-bucket-location.

[martindurant]

why don’t we use MaxKeys=0 as well

I think this was designed for sub-directories, where we want to distinguish whether it’s the actual path which exists or a sub-path. That wouldn’t apply to buckets, though. So MaxKeys could depend on the context.

[isidentical]

While this certainly adds an extra request, this does seem to be in the error-path rather than the happy-path, so it’s surely not much of a big deal?

Agree with @martindurant that it would reduce the burden, but still costful one some cases.

And I think the results are also cached?

No, we don’t cache the results.