[aws-eks] Support S3 private bucket repository from helm deployment

Use Case

Right now helm deployment only support public github repository. We need to read the repository from s3 private repo

Proposed Solution

I would like to help to develop the feature but would like to hear your opinion on the approach

1. Create a s3 presign url and use the presign url to deploy the chart new HelmChart(this, 'cortexHelm', { cluster: props.cluster, chart: props.presignUrl, namespace: props.environment.cluster.namespace, values: overrides, release: 'cortex' });

2. install S3 plugin at lambda layer: https://github.com/aws-samples/aws-lambda-layer-kubectl Use the S3 plugin to read from S3 private repo while deploy helm chart: Plugin link: https://github.com/hypnoglow/helm-s3

Other

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[nicolai86]

Filling in some details here: Let’s assume you are a CDK user which has some privately developed Helm chart which you want to deploy via eks.HelmChart construct. For simplicity you use CDK already to upload your Helm chart to S3.

You need a way to have the eks.HelmChart construct download and use these private Helm charts. Today this doesn’t work because the custom cloudformation resource which executes helm doesn’t have permissions to access this object on S3, and as a CDK user there is no way to attach the right permissions because the handler role is private.

We can make this work by introducing a helper method on eks.HelmChart, fromAsset, which grants the kubectl handler role the permissions to access the object on S3. Because permissions are created as part of the eks.HelmChart.fromAsset method we have to ensure these permissions are narrow.

In code, this would look something like this:

new eks.HelmChart(this, 'private-helm-chart', {
  chart: eks.HelmChart.fromAsset('path/to/asset.tgz')
});

Under the hood, fromAsset will create a CDK Asset and attach s3:GetObject and kms:Decrypt to the kubectl handler role. We need to change the helm handler in python slightly, like this:

     # "log in" to the cluster
    subprocess.check_call([ 'aws', 'eks', 'update-kubeconfig',
        '--role-arn', role_arn,
        '--name', cluster_name,
        '--kubeconfig', kubeconfig
    ])

    # download helm chart if hosted on s3
    if (chart is not None) and (chart.startswith('s3://')):
        bucket_name_key = chart.split('//')[1]
        bucket_name = bucket_name_key.split('/')[0]
        object_key = '/'.join(bucket_name_key.split('/')[1:])
        local_path = '/'.join(('', 'tmp', os.path.basename(object_key)))
        s3 = boto3.client('s3')
        with open(local_path, 'wb') as f:
            s3.download_fileobj(bucket_name, object_key, f)
        chart = local_path

It’s important to use the CDK Asset construct here. First we want the helm chart file to be uploaded automatically when cdk deploy is executed. Next the uploaded Helm chart must be immutable, or we risk breaking deployments/ rollbacks. Assets help here because they generate names, containing the hash of the source file.

I think it’s a relatively small change which feels idiomatic as CDK has other classes which also provide fromAsset methods. When downloading the file inside the existing code we also don’t need custom lambda layers to begin with, and we don’t generate a pre-signed url either.

What are your thoughts here, @eladb ? Will this provide a good user experience while ensuring that CDK internally generates narrow & secure IAM policies?

[polothy]

Not sure if this makes anything easier, but ECR now support OCI: https://aws.amazon.com/blogs/containers/oci-artifact-support-in-amazon-ecr/

Helm can then deploy a chart that is in ECR.