How to use AWS AssumeRole with MFA

Sorry, this is not really a bug nor a feature request.

My colleague is experimenting with aws-vault so we can enforce MFA for our cluster. He has a aws config file like this:

[profile xx-xx-prod]
role_arn=arn:aws:iam::xx:role/xxAdminAccessForLogin
source_profile=xx-login
mfa_serial=arn:aws:iam::xx:mfa/XXXXXXXXXXX
region=eu-west-1

[profile xx-xx-staging]
role_arn=arn:aws:iam::xx:role/xxAdminAccessForLogin
source_profile=xx-login
mfa_serial=arn:aws:iam::xx:mfa/XXXXXXXXXXXXXXXXXX
region=eu-central-1

[profile xx-login]
#yes, nothing here

He can then use this to do terraform commands and for example start Lens through: aws-vault exec xx-xx-staging -- kontena-lens --context=arn:aws:eks:eu-central-1:xx:cluster/xx-xx-staging

However I don’t really like this as this prevents the option from having multiple clusters open in one Lens window. You would need to restart it every time.

So: then I found Leapp. If I understand it correctly this should solve the problem described above. However I don’t really understand how to set it up.

First I add a session for the ‘login’ account:

When I start that session I get asked for a MFA serial and then it succeeds.

Then I want to add the account I use to “AssumeRole”:

I can also create/start this session with no problem. I do see credentials in ~/.aws/credentials. However when I want to run a terraform apply for example I get this error:

╷
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ 
│ 
╵
╷
│ Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: NoCredentialProviders: no valid providers in chain
│ caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
│ SharedCredsLoad: failed to load profile, .
│ EC2RoleRequestError: no EC2 instance role found
│ caused by: RequestError: send request failed
│ caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
│ 
│ 
│   with module.shared.provider["registry.terraform.io/hashicorp/aws"],
│   on ../shared/main.tf line 41, in provider "aws":
│   41: provider "aws" {
│ 

I cannot post the entire terraform files but I think this is the relevant part:

terraform {
  backend "s3" {
    bucket  = "xx-terraform-state-xx-staging"
    key     = "network/terraform.tfstate"
    region  = "eu-central-1"
    profile="xx-xx-staging"
  }

  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "3.72.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
    profile="xx-xx-staging"
}

If I remove the profile from the terraform files and run init I get:

│ Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│       For verbose messaging see aws.Config.CredentialsChainVerboseErrors

So I am hoping that there are some people here that have some experience and can give some pointers. Can we even use Leapp to solve the problem as described? I’m I using the wrong config? I don’t really understand how it should work.

edit: incuded the wrong error

[Mattie112]

Yeah not sure, for now I can also just remove the profile from the terraform files and just use the AWS_PROFILE env variable it works perfectly with that.

Also tested it today on Linux/Ubunbu seem to be working fine. Will be using it for some time and report back what I find 😃

[Nuru]

@Mattie112 “Assume Role” means different things in different contexts. I’m not quite sure I know what you are talking about, so I will just give a short primer on the whole topic.

Whether or not you are using the aws CLI or other tools that work with AWS, credentials are handled relatively consistently via the AWS SDK (published as a library in several programming languages and thereby incorporated into most programs designed to work with AWS). Once you configure the AWS CLI correctly, all the other tools should work, too.

The $AWS_CONFIG_FILE (by default $HOME/.aws/config) contains profiles which specify (among other things) what AWS IAM Role to assume when using that profile and where to get the credentials that allow you to use that profile. I categorize these profiles into what I call “Primary” and “Derived” profiles. (I don’t think AWS has terminology for this, so I invented my own. If they do and someone would educate me about it, I will use theirs if it is not too confusing.)

First, lets go over everything without MFA:

• A Primary profile has credentials provided directly. When you (heaven forbid!) add AWS access keys into the profile or (much more securely) log in via Leapp, Leapp provides the credentials for the profile you specify in the Leapp session. aws-vault does the same kind of thing. (AWS SSO does, too, although a bit differently, and usually administrators of AWS SSO do not configure it in a way that lets you “assume” roles but instead configure it so that you can log into any of a number of Primary profiles, so check with your administrator if you are logging in via AWS SSO.) The main thing is that a Primary profile does not have a source_profile entry.
• A Derived profile does have a source_profile entry. It does not get credentials on its own, but instead it gets credentials from the source profile it references. To use a derived profile, the source profile has to have valid credentials, and it has to be allowed to assume the role specified in the Derived profile. Note that it is not a requirement that the source profile be a Primary profile. The source profile can be another Derived profile; all that matters is that the chain of source profiles end at a role with valid credentials and that at every step in the chain, the source profile is allowed to assume the role specified in the Derived profile that references it.

You should only use Leapp (or aws-vault) to log into Primary profiles, and it is not required to have your Primary profiles included in your $AWS_CONFIG_FILE. The AWS SDK provides a separate file, $AWS_SHARED_CREDENTIALS_FILE (by default $HOME/.aws/credentials) where Leapp and other credential providers put the required profile information, including session credentials. Once logged into a Primary profile, any tool should be able to use that profile or any Derived profile simply by specifying which profile to use (usually by setting the AWS_PROFILE environment variable via export AWS_PROFILE=profile-name or via a command line flag, configuration string, or configuration file, such as in the case of kubectl via the users.user.exec.env section of the $KUBECONFIG file).

Just as you might not want to put your Primary profiles in your $AWS_CONFIG_FILE, I recommend you do not put any of your Derived profiles in Leapp. You might want to have your Primary profile in your $AWS_CONFIG_FILE so you can set CLI defaults or your default AWS region, but I can think of no reason to put Derived profiles in Leapp. Put them in the standard $AWS_CONFIG_FILE and they should work fine automatically, guarded by your login/logout of your Primary profile.

Adding MFA

Now, best practice, you want everything protected by MFA. Great, easy, Leapp has you covered (except for not allowing you to edit sessions yet ☹️) Re-Create your Primary profile in Leapp, this time configuring in your MFA device. (Logging in via SAML or AWS SSO? Don’t worry, as long as they are requiring MFA, you’re good.)

Now everything should magically work as before, except every once in a while Leapp will prompt you for a new MFA token.

There is one catch. The AWS documentation and examples are confusing. In this scenario it is important that you do not include mfa_serial in the Derived profile configuration. The use case for having mfa_serial in the Derived profile is if you did not use MFA to authenticate to your Primary profile. In that rare case, the Derived profile really turns into a sort of Primary profile, because it is responsible for providing a credential (the MFA token) that the source profile does not have. That is when you need mfa_serial in the Derived profile and when you should put it in Leapp instead of just the $AWS_CONFIG_FILE.

If you put mfa_serial in your Derived profile, you will repeatedly and needlessly be prompted for an MFA token every time you assume the role, which, because it is a chained role, will be at least once an hour, and that prompt may not be visible because it is coming from the AWS SDK. With your Primary profile you only get prompted when your session expires, which can be as long as 12 hours.

OP’s specific case

In your case, @Mattie112:

• remove mfa_serial from xx-xx-prod and xx-xx-staging
• remove xx-xx-prod and xx-xx-staging sessions from Leapp
• Log into xx-login via Leapp with MFA

Pretty much everything else should work. You control what role is active via the $AWS_PROFILE shell environment variable or other “profile” setting available depending on the command/application you are using.