Implement _get_kwargs_from_urls

Using URLs is very convenient, it allows configuring the storage via a single environment variable.

However S3FileSystem does not implement _get_kwargs_from_urls so it is impossible to specify anything via the URL, only the bucket.

It would be good to have S3FileSystem parse query arguments, such as endpoint_url (useful for testing locally with Minio for instance), region_name, requester_pays, signature_version, and timeouts.

Right now I have to do my own parsing on the URL to pass it to S3FileSystem, which duplicates the effort in fsspec.

[DavidMStraub]

I think this is a great idea.

I bumped into this thread while trying to find out if I can specify the endpoint_url when using the s3fs implementation in pandas, when doing things like pd.from_csv("s3://..."). Given the proliferation of S3-compatible object storage provides, I think this is an important use case. Unfortunately, the pandas implementation does not allow to pass any arguments to S3FileSystem.

Moreover, I found a recent PR in pandas implementing this via environment variables (https://github.com/pandas-dev/pandas/pull/29050) and it was closed by the panda devs arguing that this should be implemented in s3fs instead.

I think making these options configurable via the URL string would be the perfect solution.

[martindurant]

Note that following https://github.com/pandas-dev/pandas/pull/34266 , it should become possible to pass parameters to the filesystem backend sometime in the future.