Unable to create a directory

What happened: Trying to create a directory in a bucket and it silently fails.

What you expected to happen: Expect a key created in a bucket.

Minimal Complete Verifiable Example:

from s3fs import S3FileSystem
s3 = S3FileSystem(anon=False)
s3.makedirs('s3://my-bucket-name/testdir', exist_ok=False) #fails to create 
s3.makedirs('my-bucket-name/testdir', exist_ok=False) #fails to create

How do I create a directory in a bucket? Am I doing something wrong or is it a bug? I see conversation in #245 and I was wondering if you could please help explain what the intended behavior is.

Anything else we need to know?: s3fs version == 0.5.0 Relevant issue here: #245

Environment:

• Dask version:
• Python version: 3.7.7
• Operating System: Ubuntu Linux
• Install method (conda, pip, source): conda

[martindurant]

This is a long-standing misunderstanding, and your help, @srib , would be greatly appreciated in clarifying the situation for other users.

s3 does not support directories, except the top-level buckets, which are fairly similar in their behaviour. If you expect s3.makedirs('my-bucket-name/testdir') to enable you to write files with paths that start with that string then: congratulations! There is in fact nothing for s3fs to do, the location was already writable.

Once you have a file like “my-bucket-name/testdir/testfile”, the directory will appear to exist, without that file it will not. This is similar to trying to commit an empty directory to git.

Unfortunately, the situation is complicated by the behaviour of the s3 console, which has a “make directory” button, whose action is actually to create an empty file with the name of the directory you wanted. This is not part of the s3 API, just a convention, and the AWS CLI does not do this. If we were to do this, we end up making extra requests, and have to distinguish between real common preixes (directories implied by having contents) and empty placeholder files.

[martindurant]

That would make sense if mkdir were only called by end users, but other libraries call it internally. Since the intent there is “make sure that files can be written as children of this path”, the co-op does exactly the right thing - and errors and warnings would just get in the way.