S3: provide java.nio.FileSystem implementation

Expected Behavior

The java.nio.FileSystem API provides an abstraction for dealing with different types of file systems, and for accessing files and folders within that file system. AWS should provide an implementation of this interface backed by an s3 client.

Current Behavior

Application code either has to explicitly know about s3 (by passing around S3Client everywhere) or else use some custom abstraction, which inevitably ends up being a half-baked implementation of parts of the FileSystem and Path APIs anyway.

Possible Solution

There are two existing attempts to solve this problem that I can find:

• Upplication/Amazon-S3-FileSystem-NIO2

  • built against the old 1.11 AWS SDK, so incompatible with codebases that use aws-sdk-java-v2
  • no code changes, and very little activity in issues, for over a year so not actively maintained?

• elerch/Amazon-S3-FileSystem-NIO2

  • fork of the Upplication implementation, but switches to aws-sdk-java-v2
  • however this largely appears to be a solo effort, no guarantees about support, bugfixes etc going forwards

• carlspring/s3fs-nio

  • newly-created (as of November 2020) which aims to consolidate all the others
  • see comments on this thread

Fortunately, this code is MIT-licensed so perhaps could form the basis of an official library?

Context

Developing application code against s3 can be problematic because it’s not always possible to have a live s3 instance available: access policies might be very strict (no access to s3 from outside corp network, or only from specific (non-dev) machines), or developers might not even have internet access at all when e.g. working on the move.

Attempts to solve this problem in a different way exists (e.g. libraries which provide a local webserver with an s3-like interface) but this adds another set of dependencies, another tool to have to learn / configure / debug.

In my view, a cleaner solution would be to provide an implementation of java.nio.FileSystem which is the standard Java abstraction for dealing with different file systems. Application code would only need to talk to java.nio.file.Path and friends and developers could be confident of their code working reliably regardless of whether it’s running against local disk storage, or s3.

[carlspring]

Hi guys,

We are pleased to let you know that we’ve created a spin-off project (rebranded fork called s3fs-nio) of Upplication/Amazon-S3-FileSystem-NIO2. This is a spin-off (of the latest Upplication/Amazon-S3-FileSystem-NIO2 master with fully preserved history), instead of just another fork, as the upstream has a plethora of forks which contain fixes for bits and pieces that had pull requests against the upstream which were never merged. Most of these forks appear to have sadly died out, just like the upstream seems to have died (Upplication/Amazon-S3-FileSystem-NIO2#135).

As there is a need for such a library, we have decided to take on this task and rebuild an active project with a knowledge base, chat channel and helpful community around it. Ultimately, we would really appreciate it, if we could have some of the Amazon folks helping out with advice and reviewing pull requests, as this would be a massive help for us!

We’ve done a big clean up of the code, upgraded its depenencies and migrated to AWS SDK v2 (special thanks to @ptirador for all the hard work, as well as to @elerch + @markjschreiber for their advice and reviews!). The project is actively tested against JDK 8 and 11 via Github Actions. Our work is not done and we would like to keep working on this project! We intend to invite all the contributors with open pull requests to join our efforts and forward-port their fixes to our project.

If anyone is interested in lending a hand and joining our project, please reach out, as we have plenty to do! 😃

@ashleymercer : Could you please add us to your list above? Thanks! 😃

cc: @ptirador , @steve-todorov , @sbespalov

[lbergelson]

@millems This would be very useful for us. The proliferation of forks of the Upplication provider ( which is itself a fork of an older provider) causes a lot of confusion.

Google has a very robust open source Path provider for gs: buckets. It would be great if Amazon did too.