Local dataset server like MinIO [FEATURE]

🚨🚨 Feature Request

• Related to an existing Issue

Is your feature request related to a problem?

In my environment, there are multiple local servers storing data and training models. To properly manage data, I wanna isolate one machine and start a data server like MinIO. Unfortunately HUB can only store data locally, or upload to S3, Google and Azure. Is there a plan to work on a local server?

If your feature will improve HUB

To make HUB work on a collaborative team with local shared data.

Description of the possible solution

Adding more options to url and token,.

[gabriel-milan]

I was also thinking about it, and then I found this test case where you can set url to s3://whatever and the token to something like this:

token = {
    "aws_access_key_id": "your_minio_access_key",
    "aws_secret_access_key": "your_minio_secret_key",
    "endpoint_url": "your_minio_url:port",
}

and it just works! I’ve tested myself by hosting a local minio/minio container and this Getting started example.

[AbhinavTuli]

Hey @qizhen816, hub should already work with MinIO, take a look at this example for reference. Do let me know if you face any issues.