Setting a custom endpoint_url

How do I pass a custom endpoint url to s3fs.S3FileSystem ?

I’ve tried:

kwargs = {'endpoint_url':"https://s3.wasabisys.com",
          'region_name':'us-east-1'}
self.client = s3fs.S3FileSystem(key=AWS_ACCESS_KEY_ID, 
                                secret=AWS_SECRET_ACCESS_KEY,
                                use_ssl=True,
                                **kwargs)

However I get the error:

  File "s3fs/core.py", line 215, in connect
    **self.kwargs)
TypeError: __init__() got an unexpected keyword argument 'endpoint_url'

I’ve also tried passing kwargs as the parameter config_kwargs and s3_additional_kwargs with similar errors. The problem with these is that endpoint_url gets passed to botocore.Config when it shouldn’t.

I can verify boto3 is working with the following:

client = boto3.client("s3",
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
        endpoint_url="https://s3.wasabisys.com",
        use_ssl=True,
        region_name="us-east-1", 
        api_version=None,verify=None, config=None)

fs-s3fs also works

[martindurant]

Watch this space! Once https://github.com/pandas-dev/pandas/pull/34266 is in, we will be adding storage_parameters= to the various pandas IO functions, as is already available in the Dask versions.

[martindurant]

@npezolano , since you’ve just looked through the documentation, you may be the best placed to consider where this information should appear. Would you be willing to provide a PR?