[Feature Request] Support for s3fs

🚀 Feature Request

Hi, I’ve been using Hydra for 6 mos or so, so this is more of a question than a feature request, since I don’t know for sure that this isn’t already supported. I’d like to modify hydra.run.dir to an s3 path (e.g. s3://<bucket_name>/<output_dir>) and then have Hydra write all outputs to that s3 bucket. It seems like if a user already has AWS CLI configured, then (maybe?) this functionality wouldn’t be too hard to support using something like the s3fs package.

Motivation

I’d like to run my program in AWSBatch, so I need a way to pipe any output of the program to S3, similar to how Hydra already captures all output in a single directory.

Pitch

What would be ideal for me as a user would be for Hydra to automatically determine whether to use the local filesystem or s3fs, based on whether I override hydra.run.dir with a directory with an s3:// prefix.

The immediate alternative would be to periodically copy the contents of the output directory to S3 within my program.

I’d be willing to open a pull request but I would probably need some guidance on how to get started.

[omry]

Hi @samuelstanton! I have plans to abstract the working directory in Hydra, probably in 1.1. This will open up the path for a plugin that will support s3 as the working directory backend.

I am keeping it open as a reminder for the desire for s3 support for it. In the short term you are on your own. one thing you can consider is mounting your s3 bucket locally using something like s3fs-fuse and configure the Hydra sweepr/run dir to that mount point.

[oliversssf2]

Maybe consider adopting fsspec so that not only S3, Azure, GCS, and many more file systems can also be used?