multiple aws accounts with chained providers

We have a mutliple aws account setup. In the external accounts we need to modify resources, e.g. establish vpc peerings or add routes to a transit gateway.

There is a specific role in each account containing permissions to execute those external resource modifications. To assume those roles in the external accounts, there is one role called intermediateRole that is allowed to assume those roles in the external accounts.

To maintain this multi aws account setup with pulumi we tried to establish chained providers with the according roles. We experience issues with one particular external account/role while the other one works.

Steps:

1. create a single intermediateProvider by assuming the intermediateRole
2. create a destinationProvider by assuming the external role for controlling resources in an external aws account, based on the intermediateProvider
3. create anotherDestinationProvider by assuming another external role for controlling resources in another external aws account, based on the intermediateProvider

using the provider from step 2, works as expected using the provider from step 3, though similar build up to the one from step2, fails. expressed via code:

An example of our functions:
pulumi.Run( ..
  intermediateProvider, err := aws.NewProvider(ctx, "intermediateProvider", &aws.ProviderArgs{
    AssumeRole: &aws.ProviderAssumeRoleArgs{
      RoleArn:     intermediateRole.Arn,
      SessionName: pulumi.String("intermediateProviderSession"),
    },
    Region: pulumi.String(region.Name),
  })
  if err != nil {
    return err
  }
  
  // This code path works
  createTGWPeering(ctx, intermediateProvider, ...)

  // This code path fails
  for _, account := range peeringAccounts{
    peerAccounts(ctx, account, intermediateProvider, ....)
  }

whithin each function the correlated chained destinationProvider are built the same way:

  createTGWPeering(ctx *pulumi.Context, ...) {
		destinationProvider, err := aws.NewProvider(ctx, "destinationProviderProvider", &aws.ProviderArgs{
			AssumeRole: &aws.ProviderAssumeRoleArgs{
				RoleArn:     pulumi.String(fmt.Sprintf("arn:aws:iam::%s:role/%s", AccountID, roleName)),
				SessionName: pulumi.String("destinationProviderSession"),
			},
			Region: pulumi.String(region.Name),
		}, pulumi.Provider(intermediateProvider),
		)
		if err != nil {
			return err
		}
		assumed, error := aws.GetCallerIdentity(ctx, nil, pulumi.Provider(destinationProvider))
		if error != nil {
			fmt.Printf("assuming role failed: %s \n ", error)
			return error
		}
		fmt.Printf("assumed role via destination provider %s \n", assumed.Arn)
  }

peerAccounts(ctx *pulumi.Context, ...) {
		anotherDestinationProvider, err := aws.NewProvider(ctx, "destinationProviderProvider", &aws.ProviderArgs{
			AssumeRole: &aws.ProviderAssumeRoleArgs{
				RoleArn:     pulumi.String(fmt.Sprintf("arn:aws:iam::%s:role/%s", anotherAccountID, anotherRoleName)),
				SessionName: pulumi.String("anotherDestinationProviderSession"),
			},
			Region: pulumi.String(region.Name),
		}, pulumi.Provider(intermediateProvider),
		)
		if err != nil {
			return err
		}

		anotherAssumed, error := aws.GetCallerIdentity(ctx, nil, pulumi.Provider(anotherDestinationProvider))
		if error != nil {
			fmt.Printf("assuming another role failed: %s \n ", error)
			return error
		}
		fmt.Printf("assumed role via another destination provider %s \n", anotherAssumed.Arn)
	}

the aws.GetCallerIdentity calls are exemplary for using the created providers. For the latter call we get the error:

error in peering sts identity: rpc error: code = Unknown desc = invocation of aws:index/getCallerIdentity:getCallerIdentity returned an error: 1 error occurred: * error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::[anotherAccountID]:role/[anotherRoleName]) cannot be assumed. There are a number of possible causes of this - the most common are: * The credentials used in order to assume the role are invalid * The credentials do not have appropriate permission to assume the role * The role ARN is not valid Error: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

at this point I came to the conclusion that the role setup in the latter external aws account is wrong hence I tried to verify it via the aws cli, which worked fine. Therefore I tried a workaround with aws go sdk to directly assume the intermediate role and create the failing destinationProvider:

	intermediateRole.Arn.ApplyT(func(roleArn string) string {

		cfg, err := config.LoadDefaultConfig(context.TODO())
		if err != nil {
			panic("configuration error, " + err.Error())
		}

		client := sts.NewFromConfig(cfg)

		session := "intermediateSession"
		input := &sts.AssumeRoleInput{
			RoleArn:         &roleArn,
			RoleSessionName: &session,
		}

		result, err := client.AssumeRole(context.TODO(), input)
		if err != nil {
			fmt.Println("Got an error assuming the role:")
			fmt.Println(err)
		}

		anotherDestinationProvider, err := aws.NewProvider(ctx, "AnotherDestinationProvider", &aws.ProviderArgs{
			AccessKey: pulumi.String(*result.Credentials.AccessKeyId),
			SecretKey: pulumi.String(*result.Credentials.SecretAccessKey),
			Token:     pulumi.String(*result.Credentials.SessionToken),
			AssumeRole: &aws.ProviderAssumeRoleArgs{
				RoleArn:     pulumi.String(fmt.Sprintf("arn:aws:iam::%s:role/%s", anotherAccountID, anotherRoleName)),
				SessionName: pulumi.String("AnotherSessionProvider"),
			},
			Region: pulumi.String(region.Name),
		},
		)
		if err != nil {
			fmt.Println("peering provider failed", err)
		}

which worked fine out of the box. How can I debug this further? Right now I cannot see any difference, that would explain the inconsistent behaviour of the chained providers.

Expected: the intermediate pulumi provider should work for all chained destinationProviders Actual: it doesn’t work for all

[leezen]

I don’t think the Python example you have above is functionally equivalent to your Go code as you don’t actually construct the peering provider using the intermediate provider as a resource option. The Go example you have does look like what I’d expect and we’ll need to do some investigation to understand what’s going on there.

[leezen]

Thanks for the additional details. Digging into this further, it looks like this is an upstream issue as tracked in https://github.com/hashicorp/terraform-provider-aws/issues/16841