Proposal: S3 Remote should respect AWS_PROFILE env var if no profile provided in DVC config

I believe that the DVC CLI should utilise the AWS_PROFILE env var when a profile is not provided for a remote

For example, after setting the AWS_PROFILE:

$ export AWS_PROFILE=someprofile

The following command works:

aws s3 ls s3://mybucket

But this command does not:

dvc pull

Similarly, it’s common to set a profile for a one off command:

AWS_PROFILE=someprofile aws s3 ls s3://mybucket

DVC should be invokable in a similar way:

AWS_PROFILE=someprofile dvc fetch

It’s worth noting that DVC already has some implicit behaviour when no profile is provided, it will default to the profile named default which matches the AWS CLI behaviour.

I propose that at this point, DVC should check whether AWS_PROFILE env var is set and use that as the Profile name. If AWS_PROFILE is unset or blank, then default would be used

This means that any aws profiles explicitly set in the DVC config will be respected before using the AWS_PROFILE env var.

Happy to implement this one but wanted to get feedback on it before writing any code

[pmrowla]

AWS_PROFILE is already handled and supported in the underlying botocore library used to interact with AWS (and the env var does not need to be read at the DVC source level).

Do you have use_accelerate_endpoint set in your AWS config? It’s possible that your problem is related to https://github.com/iterative/dvc/issues/6588

Setting profile = ... in your DVC remote config (instead of using AWS_PROFILE) does change how and where the profile is loaded (which may be why your workaround works, and is potentially what causes the problem in #6588)

[pmrowla]

@diesal11 it sounds like there is some other issue going on, AWS_PROFILE should already be respected in DVC

can you post the full error log from dvc pull -v