Trouble connecting to S3

I was recently trying to read content with acl-private from S3 and had trouble with dask.bag.read_text. Here are some of the step I took. In the end, I was still unable to connect. Note I’m using an IAM account, but I did verify separately that my account has S3 full access:

Example: The code below would raised a NoCredentials error.

import dask.bag as db
b = db.read_text('s3://xdata-breach-data/dumps/*.txt',
                 errors='ignore', 
                 blocksize=10000000)

In the case of the workers, I have a .aws/credentials file on each node. I was also able to verify that both boto and boto3 were able to find credentials:

import boto
print(boto.connect_s3()) # ----> no problem

import boto3
print(boto3.client('s3')) # ----> no problem

I configure via the following methods config file methods:

• .boto
• .aws/config
• .aws/credentials
• /etc/boto.cfg
• environment variables in /etc/profile and .bashrc

I also attempted passing the credentials via read_text->storage_options. In this case I got TypeError: __init__() got multiple values for argument 'aws_secret_access_key'

[martindurant]

The arguments provided in storage_options should match the ones that successfully list the files:

storage_options={'client_kwargs': {'endpoint_url':'https://os.zhdk.cloud.switch.ch'}, 'anon':True}

[martindurant]

Glad to hear!