question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(pipelines): DockerCredential.docker_hub() silently fails to auth against Docker Hub

See original GitHub issue

Using DockerCredential.docker_hub() from CDKPipeline will silently fail to auth against Docker Hub during all stages of the build. In our case this makes ContainerImage.from_asset() intermittently fail when executed via CDKPipeline and makes the overall CI/CD process unstable.

Because most users rely on public images on Docker Hub the failure mode is subtle. In most cases users will observe only intermittent issues due to Docker Hub enforcement of pull rate limits against the IP addresses in AWS CodeBuild infrastructure.

To reliably reproduce this issue one can create a free Docker Hub account and create a private container image. Construct a pipeline that uses from_asset() to build an image that depends on the private image. Finally, supply a DockerCredential structure to CDKPipeline that is expected to allow access to the private image. This approach will reliably demonstrate that the Docker Hub credentials are not being used.

Private images on Docker Hub are an unusual use case; however using Docker Hub credentials to pull public images from Docker Hub is common practice due to Docker Hub enforcing IP address rate limits for unauthenticated users. The IP address rate limits are especially painful when building on shared infrastructure such as AWS CodeBuild.

Docker Hub offers per-user rate limits for free accounts and it is common practice to setup free accounts to support automated builds. When using shared infrastructure such as AWS CodeBuild, Docker Hub credentials are a practical necessity.

Reproduction Steps

dockerhub_secret = secretsmanager.Secret.from_secret_name_v2(
        self,
        "pull-credentials",
       "dockerhub/image-pull-credentials"
)

pipeline = CdkPipeline(
            self,
            "CICDPipeline",
            docker_credentials=[
                pipelines.DockerCredential.docker_hub(dockerhub_secret),
            ],
...

Because the common failure mode is caused by rate limit enforcement on Docker Hub I cannot provide a copy/paste test that will reliably demonstrate the problem without also sharing valid Docker Hub credentials which I do not wish to do.

What did you expect to happen?

I expected the container asset stages in AWS CodeBuild to authenticate against Docker Hub when pulling image dependencies.

By authenticating I expected to be subject to a per-user pull rate limit on Docker Hub despite the build happening on shared infrastructure in AWS CodeBuild.

What actually happened?

The builds intermittently fail due to enforcement of IP address rate limits at Docker Hub.

Environment

  • CDK CLI Version : 1.115.0
  • Framework Version:
  • Node.js Version: v14.17.2
  • **OS : AWS CodeBuild (Amazon Linux) **
  • Language (Version): Python 3.9.6

Other

Authentication fails because credHelpers in Docker’s config.json does not work as the Docker documentation would lead one to believe.

When pulling a public image (e.g. FROM amazon/aws-cli:2.2.18) the lookup in credHelpers is for the key https://index.docker.io/v1/. When the credential helper program is executed the domain passed to the helper is index.docker.io. The CDK incorrectly presumes that both values are index.docker.io so the credential helper is never invoked. Because most images on Docker Hub are public the lack of authentication is not obvious except when pull rate limits are exceeded.

This is a workaround to the problem:

        pipeline = CdkPipeline(
            self,
            "CICDPipeline",
            docker_credentials=[
                pipelines.DockerCredential.docker_hub(dockerhub_secret),
                pipelines.DockerCredential.custom_registry("https://index.docker.io/v1/", dockerhub_secret),
            ],

The above workaround creates two very similar entries, one to support each stage of the auth process.

The entry for https://index.docker.io/v1/ is required to cause docker to invoke the CDK credential helper.

The entry for index.docker.io is required for the CDK credential helper to find a match for the domain passed into the credential helper.


This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:4
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
mpetito-envativecommented, Oct 14, 2021

This workaround doesn’t appear to be working. I still receive dockerhub rate limit errors in the CDK pipeline synth step after setting up a paid Dockerhub account and corresponding secret configuration in AWS.

var dockercreds = Amazon.CDK.AWS.SecretsManager.Secret.FromSecretNameV2(this, "DockerhubSecret", "dockerhub");

var pipeline = new CodePipeline(this, "Pipeline", new CodePipelineProps
{
    ...
    DockerEnabledForSynth = true,
    Synth = new ShellStep("Synth", new ShellStepProps
    {
        ...
    }),
    DockerCredentials = new[]
    {
        DockerCredential.DockerHub(dockercreds),
        DockerCredential.CustomRegistry("https://index.docker.io/v1/", dockercreds) // https://github.com/aws/aws-cdk/issues/15737
    }
});

I’ve verified that the IAM Role that CodeBuild is using has access to the secret. (Note: the originally generated IAM policy did not have access because the secret arn was suffixed with dockerhub-????? for some reason.)

I also get the following output in the synth step as expected:

[Container] 2021/10/14 14:15:55 Running command echo '
{
    "version": "1.0",
    "domainCredentials": {
        "index.docker.io": {
            "secretsManagerSecretId": "arn:aws:secretsmanager:us-east-1:<account-id>:secret:dockerhub"
        },
        "https://index.docker.io/v1/": {
            "secretsManagerSecretId": "arn:aws:secretsmanager:us-east-1:<account-id>:secret:dockerhub"
        }
    }
}
' > $HOME/.cdk/cdk-docker-creds.json

The asset that fails during synth is built using the node:lts docker image as part of an S3 bucket deployment.

var source = Source.Asset(..., new Amazon.CDK.AWS.S3.Assets.AssetOptions
{
    Bundling = new BundlingOptions
    {
        Image = DockerImage.FromRegistry("node:lts"),
        Command = new[]
        {
            "bash", "-c", string.Join(" && ", new []
            {
                "npm ci",
                "npm run build",
                "cp -r /asset-input/build/* /asset-output"
            })
        }
    },
    ...
});
0reactions
github-actions[bot]commented, Jan 10, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

Read more comments on GitHub >

github_iconTop Results From Across the Web

docker login - Docker Documentation
When you log in, the command stores credentials in $HOME/.docker/config.json on Linux or %USERPROFILE%/.docker/config.json on Windows, via the procedure ...
Read more >
class DockerCredential · AWS CDK
Represents credentials used to access a Docker registry. Example. const dockerHubSecret = secretsmanager.Secret.fromSecretCompleteArn(this, 'DHSecret', ...
Read more >
docker login - error storing credentials - write permissions error
The Problem: you try to login into docker registry and unable to authenticate due to storing credentials error.
Read more >
Using Docker with Pipeline - Jenkins
Since containers are initially created with "clean" file systems, this can result in slower Pipelines, as they may not take advantage of on-disk...
Read more >
Customize containers with Databricks Container Services
Learn how to create Databricks clusters using a custom Docker image, for full control of library customization, environment lockdown, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found