Error obtaining subject token for workload identity federation through AWS

Environment details

• Programming language: C#
• OS: Controlled by AWS (lambda function)
• Language runtime version: 6.0
• Package version: Google.Apis.Auth 1.58.0-beta01

Following issue #2033 I was trying to use identity federation from AWS to GCP. The idea was to assume a role in AWS and use this role’s credentials to call GCP through identity federation. But I couldn’t get the solution to work and always end-up with this error message.

Not sure if it’s because of something in my implementation or a bug in the SDK because we could get similar flow to work in python but not in .NET. I included the code I used below. Appreciate your assistance.

Code

    [LambdaSerializer(typeof(JsonSerializer))]
    public async Task<IResult> Handle(ILambdaContext context)
    {
        Environment.SetEnvironmentVariable("AWS_DEFAULT_REGION", "eu-west-1");
        var options = ServiceProvider.GetRequiredService<AWSOptions>();
        var configuration = ServiceProvider.GetRequiredService<IConfiguration>();

        var credentials = await AssumeRole(options, configuration);

        var projectId = "mathem-ml-datahem-test";
        var awsCredentialsProvider = new StaticCredentialsProvider(new AwsCredentials() { Token = credentials.SessionToken, SecretKey = credentials.SecretAccessKey, AccessKey = credentials.AccessKeyId });
        var httpClient = new GcpAwsHttpClientFactory(awsCredentialsProvider);

        var googleCredential = (await GetGcpCredentials(configuration));

        var bigQueryClient = await new BigQueryClientBuilder()
        {
            Credential = googleCredential,
            HttpClientFactory = httpClient,
            ProjectId = projectId,
            ApplicationName = "test-app"
        }.BuildAsync();


        var response = await bigQueryClient.ExecuteQueryAsync("<QUERY>", Array.Empty<BigQueryParameter>());
        var rows = await response.ReadPageAsync(100);
        return Results.Ok(rows.Rows);
    }

    private static async Task<Credentials> AssumeRole(AWSOptions options, IConfiguration configuration)
    {
        var awsSecurityTokenServiceClient = new AmazonSecurityTokenServiceClient(options.Region);

        var assumeRoleReq = new AssumeRoleRequest()
        {
            DurationSeconds = 900,
            RoleSessionName = $"<Session-Name>",
            RoleArn = "<ARN>"
        };

        var assumeRoleRes = await awsSecurityTokenServiceClient.AssumeRoleAsync(assumeRoleReq);

        var assumeRoleClient = new AmazonSecurityTokenServiceClient(assumeRoleRes.Credentials);
        var callerIdentity = await assumeRoleClient.GetCallerIdentityAsync(new GetCallerIdentityRequest(), CancellationToken.None);

        return assumeRoleRes.Credentials;
    }

    private static async Task<GoogleCredential> GetGcpCredentials(IConfiguration configuration)
    {
       // The credentials are loaded from AWS parameter store to the environment variables
        var gcpCredentialsJson = configuration.GetValue<string>("GcpCredentials");

        return GoogleCredential
            .FromJson(gcpCredentialsJson);
    }

// This will sign the request as described here
// https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
public class GcpAwsHttpClientFactory : HttpClientFactory
{
    private readonly ICredentialsProvider _credentialsProvider;

    public GcpAwsHttpClientFactory(ICredentialsProvider credentialsProvider)
    {
        _credentialsProvider = credentialsProvider;
    }

    protected override HttpMessageHandler CreateHandler(CreateHttpClientArgs args)
    {
        var gcpHttpClient = base.CreateHandler(args);
        var handler = new AwsSignedHttpMessageHandler()
        {
            InnerHandler = gcpHttpClient
        };

        return handler;
    }
}

Error

{
  "errorType": "SubjectTokenException",
  "errorMessage": "An error occurred while attempting to obtain the subject token for AwsExternalAccountCredential",
  "stackTrace": [
    "at Google.Apis.Auth.OAuth2.ExternalAccountCredential.GetSubjectTokenAsync(CancellationToken taskCancellationTokne)",
    "at Google.Apis.Auth.OAuth2.ExternalAccountCredential.RequestStsAccessTokenAsync(CancellationToken taskCancellationToken)",
    "at Google.Apis.Auth.OAuth2.ExternalAccountCredential.RequestAccessTokenAsync(CancellationToken taskCancellationToken)",
    "at Google.Apis.Auth.OAuth2.TokenRefreshManager.RefreshTokenAsync()",
    "at Google.Apis.Auth.TaskExtensions.<>c__DisplayClass0_0`1.<<WithCancellationToken>g__ImplAsync|0>d.MoveNext()",
    "--- End of stack trace from previous location ---",
    "at Google.Apis.Auth.OAuth2.TokenRefreshManager.GetAccessTokenForRequestAsync(CancellationToken cancellationToken)",
    "at Google.Apis.Auth.OAuth2.ServiceCredential.GetAccessTokenWithHeadersForRequestAsync(String authUri, CancellationToken cancellationToken)",
    "at Google.Apis.Auth.OAuth2.ServiceCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)",
    "at Google.Apis.Http.ConfigurableMessageHandler.CredentialInterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)",
    "at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)",
    "at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)",
    "at Google.Apis.Auth.OAuth2.Requests.ImpersonationTokenRequestExtensions.ExecuteAsync(ImpersonationRequest request, HttpClient httpClient, String url, CancellationToken cancellationToken)",
    "at Google.Apis.Auth.OAuth2.Requests.ImpersonationTokenRequestExtensions.ExecuteAsync(ImpersonationRequest request, HttpClient httpClient, String url, IClock clock, ILogger logger, CancellationToken cancellationToken)",
    "at Google.Apis.Auth.OAuth2.ImpersonatedCredential.RequestAccessTokenAsync(CancellationToken taskCancellationToken)",
    "at Google.Apis.Auth.OAuth2.ExternalAccountCredential.RequestAccessTokenAsync(CancellationToken taskCancellationToken)",
    "at Google.Apis.Auth.OAuth2.TokenRefreshManager.RefreshTokenAsync()",
    "at Google.Apis.Auth.OAuth2.TokenRefreshManager.<GetAccessTokenForRequestAsync>g__LogException|10_0(Task task)",
    "at lambda_method1(Closure , Stream , ILambdaContext , Stream )",
    "at Amazon.Lambda.RuntimeSupport.Bootstrap.UserCodeLoader.Invoke(Stream lambdaData, ILambdaContext lambdaContext, Stream outStream) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/UserCodeLoader.cs:line 145",
    "at Amazon.Lambda.RuntimeSupport.HandlerWrapper.<>c__DisplayClass8_0.<GetHandlerWrapper>b__0(InvocationRequest invocation) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/HandlerWrapper.cs:line 56",
    "at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.InvokeOnceAsync(CancellationToken cancellationToken) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/LambdaBootstrap.cs:line 176"
  ],
  "cause": {
    "errorType": "HttpRequestException",
    "errorMessage": "Connection refused (169.254.169.254:80)",
    "stackTrace": [
      "at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)",
      "at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)",
      "at Google.Apis.Auth.OAuth2.AwsExternalAccountCredential.AwsMetadataServerClient.FetchMetadataAsync(String metadataUrl)",
      "at Google.Apis.Auth.OAuth2.AwsExternalAccountCredential.AwsSecurityCredentials.<>c__DisplayClass15_0.<<MaybeFromMetadataAsync>g__BuildSecurityCredentialsEndpointAsync|0>d.MoveNext()",
      "--- End of stack trace from previous location ---",
      "at Google.Apis.Auth.OAuth2.AwsExternalAccountCredential.AwsSecurityCredentials.MaybeFromMetadataAsync(AwsMetadataServerClient metadataClient, String credentialUrl)",
      "at Google.Apis.Auth.OAuth2.AwsExternalAccountCredential.AwsSecurityCredentials.FetchAsync(AwsMetadataServerClient metadataClient, String credentialUrl)",
      "at Google.Apis.Auth.OAuth2.AwsExternalAccountCredential.GetSubjectTokenAsyncImpl(CancellationToken taskCancellationToken)",
      "at Google.Apis.Auth.OAuth2.ExternalAccountCredential.GetSubjectTokenAsync(CancellationToken taskCancellationTokne)"
    ],
    "cause": {
      "errorType": "SocketException",
      "errorMessage": "Connection refused",
      "stackTrace": [
        "at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)",
        "at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)",
        "at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)",
        "at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)"
      ]
    }
  }
}

Thanks!

[amanda-tarafa]

Is it possible to pass the AWS credentials to the Gcp client in other ways than the default environment variables?

The credential first checks the environment variables, and if not present, attempts to get AWS credentials from the environment metadata server. This, and the existence of the AWS_ACCESS_ACCESS_KEY_ID bug (already fixed, pending release) is what provoked your first error, as the wrongly named environment variable was not being found and the metadata server on the lamba environment was rejecting your the request.

Basically, there’s no way to configure the the .NET Auth library so that the lambda can run with X assumed role and the GCP authentication can happen with Y assumed role. Maybe you can configure the lambda itself to obtain the X assumed role from other than the environment variables so you can use the environment variables for Y? I’m not that familiar with AWS lambda to know whether that’s possible or not, but it might be worth exploring. The alternative is, of course, to have X and Y be the same, although it’s clear that’s not your requirement.

That said, I’m pretty certain that the Python Auth library would behave in the same way, as supporting WIF for AWS is a feature that has been implemented acroos Google Auth libraries in the same manner. I’ll bring this use case to the consideration of the Auth team as a feature request for all Google Auth libraries, but don’t expect us to be able to implement any time soon. I’m sorry about that.

And, I’m still treating the initial part of this issue as a bug, as I’ve been able to reproduce your latest error, ocasionally. I’m actively working on it.

[amanda-tarafa]

I’ve been able to reproduce the same error as you are seeing, only oncasionally. I don’t know the cause yet, I’ll update here when I know more.