Does dask-cloudprovider support private dockerhub images?

Does dask-cloudprovider support private docker-hub images (or AWS Elastic Container Registry/ECS)? I tried with no joy:

import dask_cloudprovider
from dask_cloudprovider import FargateCluster
dask_cloudprovider.__version__  # 0.2.0

cluster = FargateCluster(cloudwatch_logs_group="my_log_group",
                         cluster_name_template='test',
                         scheduler_timeout="1 hour",
                         security_groups=["A CUSTOM SECURITY GROUP"],
                         n_workers=1,
                         image="asmith26/test-dask_cloudprovider",  # points to a private dockerhub (works when made public)
                        )

[UPDATED following @martindurant comment, good idea, many thanks 😃]

[jacobtomlinson]

I’m going to reopen this as we currently do not support private image pulls in FargateCluster and it would be good to add that.

ECR is the recommended approach generally as image pulls will be faster. But there is no reason why folks shouldn’t be able to pull from other private repos.

It looks like the credentials need to be stored in secrets manager and then provided in the task definition. I would be tempted to say putting the credentials in secrets manager is out of scope for FargateCluster, but we should add a kwarg for configuring the secret name in the task definition.

https://aws.amazon.com/blogs/compute/introducing-private-registry-authentication-support-for-aws-fargate/

[jacobtomlinson]

@myan-numagic could you raise a separate issue for this as you are talking about a different cloud.