Deprecation of Docker image for Insiders

Update: the documentation explains how to self-host the Docker image with 5-10mins of effort.

---

In #2031, we paved the way to publish a Docker image that contains Material for MkDocs Insiders and is installable by opting into the GitHub Container Registry. Sadly, there are now two roadblocks ahead that make it uneconomical for me to further provide this service. This issue is meant as a discussion to find solutions, so please chip in, if you’re a user of Insiders.

Transfer cost

As long as the GitHub Container Registry is in Beta, it’s free to use for public and private repositories. However, when the Container Registry exits Beta stage, it’s expected to have the same pricing as the GitHub Package Registry:

• $ 0.25 / GB storage
• $ 0.50 / GB transfer

In the last months, we were able to squeeze the compressed size of the Docker image from 64 MB down to 27 40 MB (see https://github.com/squidfunk/mkdocs-material/pull/2436#issuecomment-799232978), which is an awesome achievement. However, still, the GB can add up quickly. If an upstream dependency like MkDocs chooses to add more dependencies, the image grows. If the image size stays at 40 MB, 1 GB equals roughly 25 downloads of the Insiders Docker image. This means that every download of the Insiders Docker image costs me 2ct. If somebody pulls an older image with a size of 64 MB, it’s 3.2ct. This doesn’t sound like much, and yes, of course Insiders are paying to get access to the repository. The problem is that I have no way to attribute downloads to certain users, I can only see the total number of downloads. This means I don’t have any control about downloads (i.e. through rate-limiting), so a user could induce thousands of downloads and my bill would go up with no possibility to mitigate it, as I cannot know who’s causing the traffic.

For comparison, assuming 1 GB of data storage and 1.000 downloads (40 GB transfer out):

• S3 (us-east-1): $ 3.54
• GitHub Package Registry: $ 20.00

Of course, S3 is object storage, but this is just for a comparison of how expensive a private GitHub Package Registry is.

Programmatic access

The number of sponsors keeps growing at a steady pace, and up to now I have added and removed collaborators manually. Today, I started automating this process, but found out that this is not possible for the Container Registry, as there are no API endpoints that allow for doing that. I really hope that GitHub will add programmatic access at some point, but currently it’s not possible.

Where to go from here

I understand that this might cause some of you trouble. I want to faciliate a discussion among the community, so we can maybe find a better solution that doesn’t incur the problem with transfer cost getting out of control. Some possible solutions:

1. Move to an alternate registry that allows for programmatic access (DockerHub?)
2. Users of the Docker container publish them on their own private registry

For 1., Amazon ECR would theoretically support more fine granular access control. This would however mean that we would need explicit account management, since we’re now out of GitHub’s realm. There’s also still the problem for how to grant access to specific members of an organization, so I’m also investigating that. Maybe a GitHub App which is granted access on an organization / user repository could be an idea for better control over configuration and access management.

I’m open for ideas!

[squidfunk]

I prototyped a new workflow and think it is very straight forward:

1. Fork the Insiders repository to your own or organization account
2. Enable GitHub Actions on the new repository
3. Create a personal access token with the write:packages scope
4. Add this token to the secrets of the fork under GHCR_TOKEN
5. Optional: install GitHub Pull App to automatically stay in-sync with upstream

The workflows have been adjusted so that forks will only invoke the publish workflow on new tags that match the #.#.# scheme. The build and documentation workflows are essentially no-ops, and can be deleted (but won’t run anyway, so no need). When a new release is issued, a new tag is created, the tag is synced (manually or via the Pull App), and the release is automatically published on the registry of the user or organization account. I think that’s pretty much it, and it should be achievable in 5-10 minutes of setup 😊 I’m currently in the process of adding this to the Insiders documentation, but I wanted to share it here if users would already want to try it out.

[hellt]

In my view, the easiest way to continue making use of docker images would be to create/distribute a workflow that paid users will be able to add in their private repo to build/push a docker image to their own private ghcr.io/dockerhub/ecr registry.

To expand on this:

1. mkdocs-material provides a image-build.yml workflow file as part of the insiders artifacts
2. a paid user creates a private repo where he adds just this workflow
3. in this private repo a user adds credentials for their container registry of choice. the provided workflow may only include ghcr.io registry as an example or all of them (dockerhub, ecr)
4. workflow can be as smart as checking if the available insiders release has already been built and skip subsequent builds, or use a daily cron job/manual_trigger for simplicity
5. at the end of the day each insider will have its own container image built from the upstream insiders code and all the potential charges will be solely on the users account (if they happen to be, which is unlikely if not overused in CIs).

On Mon, Mar 15, 2021 at 6:12 PM Fedor Korotkov @.***> wrote:

I think there are couple of things worth to notice:

1. Pulls from GitHub Actions are not affecting the storage numbers.
2. I don’t think total number of downloads represents the actual downloads. I think it might be similar to Docker Hub where a cached pull is also counted as a download.

So the actual charges might be even smaller.

IMO I think the “right” solution is for GH to provide an option to charge consumers. I’ve created a GitHub Community post https://github.community/t/charge-consumers-for-packages-traffic/168076 with the idea.

In the meantime, it will be inconvenient to build the image ourselves on every release, but it’s not a critical inconvenience. If there will a simple instruction on how to automate the process so people don’t accidentally make their images public, it will be a good option.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/squidfunk/mkdocs-material/issues/2442#issuecomment-799591583, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLKV5OI7SIFEFBUFYU2LE3TDY5YNANCNFSM4ZFFN22Q .