Explore CNCF v3 buildpacks

From @jchesterpivotal in https://github.com/jupyter/repo2docker/issues/707#issuecomment-505904267

By way of warning, what follows is hilariously biased: I’ve several times worked on two generations of buildpack technology over the past 5 years. Pride makes me defensive.

As it was related to me by a Red Hatter I asked, s2i was created largely because the previous generations of buildpack lifecycles from Heroku (v2a) and Cloud Foundry (v2b) were optimised to a rootfs+tarball target (Heroku’s term is “slug”, Cloud Foundry’s is “droplet”). That was considered unsuitable for OpenShift v3, which was an image-centric architecture.

Whereas Heroku and Cloud Foundry would meet you at code and hid the underlying container infrastructure, OpenShift would meet you at the image, so the latter (this is a personal opinion) had a business need for something like buildpacks to reduce the convenience gap.

But s2i never really found a home outside of OpenShift, while buildpacks have flourished in two massive, independent but genetically-related ecosystems.

Critically, the emergence of the v2 registry API enables features (particularly layer rebasing) that were previously impossible. In addition Google’s Container Tools team developed and maintain the google-gocontainerregistry library which allows us to perform construction and rebasing operations with or without the docker daemon. The design of CNBs takes full advantage of both of these advances.

By way of speed improvements: We have observed some Java rebuilds drop from minutes to milliseconds. We expect large-cluster rollouts to drop from dozens of hours to potentially minutes.

Edit: I should add, your reasons for moving off s2i would apply to v2a and v2b buildpack lifecycles as well. One of the motivating problems faced by both Pivotal and Heroku has been exactly this sort of combinatorial explosion; CNBs are designed to make it possible to more easily compose buildpacks developed independently of one another.

I’ve bolded the bits that I think are most relevant to us. It would be great if someone could take a look at https://buildpacks.io to see if we can base repo2docker off v3 of buildpacks. http://words.yuvi.in/post/why-not-s2i/ contains reasons why we moved off s2i (which is similar to v2 of buildpacks).

A useful test case would be to try to make:

1. A buildpack for environment.yml
2. A buildpack for install.R
3. A buildpack for postBuild
4. A buildpack for apt.txt

And then see how easy / hard it is to have a repo with any combination of these 4 files produce one single image. My rudimentary math skills tell me that there’s 4! possible combinations here (24), and we shouldn’t have to write more than 4 buildpacks…

[ltalirz]

In case it’s useful, here is a link to a recent post on the topic by Jose Diaz-Gonzalez, the lead developer of dokku, including some notes on how CNB tech differs between cloudfoundry, heroku and herokuish https://dokku.github.io/technology/comparing-buildpack-v3-to-herokuish

[samj1912]

Hello 👋 I am a maintainer on the Cloud Native Buildpacks project. Happy to answer questions and provide support if y’all decide to go forward with CNB support 😃

Just some links that might be useful in this context. You can also write CNBs in Python (if that reduces the integration barrier here). Here are some example buildpacks written in python -

https://github.com/samj1912/proc-descriptor-buildpack/blob/main/main.py https://github.com/samj1912/runtime-env-descriptor-buildpack/blob/main/main.py https://github.com/samj1912/label-descriptor-buildpack/blob/main/main.py

You can take these small pieces and compose them into a “meta-buildpack” if you want to which allows you to alias the combination of the above buildpacks in a simple-to-use wrapper - https://github.com/samj1912/project-descriptor-buildpack (this is mostly just a shell with an order file https://github.com/samj1912/project-descriptor-buildpack/blob/0978e93ab8e417a63baae9f9092c646b71685518/buildpack.toml#L14 (Note that all buildpacks here are optional so you could have 2**3 -1 valid combinations here which are automatically handled)

Link to CNB Python bindings library that was used to create these - https://github.com/samj1912/python-libcnb

Link to the official golang CNB bindings - https://github.com/buildpacks/libcnb

Our katacoda tutorials to help you get off the ground quickly (fully set up with CNB tools and an interactive walkthrough on creating a simple buildpack in bash) -

https://katacoda.com/buildpacks

---

EDIT - Here is a simple postBuild buildpack - https://github.com/samj1912/postbuild-buildpack/blob/main/main.py