Allow to use jib instance as base-image in multi-project builds

This is a follow-up on #1436 (Optimize Layering for Multi-Project Setups). In Gradle multi-project people typically build and deploy many (micro)services. Accordingly many Docker images are being built. In most cases those images will share a great amount of the techstack and only differ in the business functionality. Accordingly it would make sense to introduce a shared base layer holding third-party dependencies that are shared by all projects, whereas the individual projects only hold the delta to those base images. This could look like:

dependencies{
  compile project(':service-base')
  jibBaseImage project(':service-base')
}

whereas service-base holds all the third-party dependencies necessary for the various projects. This would trigger for a service project:

• use of service-base as base layer of the Docker image (no need to specify one)
• exclusion of all dependencies of :service-base in the built image.
• check whether there is any dependency version mismatch whereas the service changes the version of a third-party library => services should not be able to overrule the version of the base image to avoid having the same JAR twice in the image.
• the jibTasks like jib setup proper dependencies.
• the jibTasks access the cache directory of the service-base to fetch the built artifacts.

The jib instance in service-base would be a regular jib instance.

Layering of Docker images has been discussed in other tickets like https://github.com/GoogleContainerTools/jib/issues/403. While a simple/pragmatic solution has been implemented, it does fully address the brought up issues that desires to have more control of the layering. With this ticket here rather advanced layering can be setup transparently in a traditional “Gradle/Maven-way” without having to deal with inner workings of Docker layering or having to manually specify which third-party libraries go into which layer.

as a further benefit the multiple jib instances automatically will be “in-sync” in terms of caching, whereas the actual base layer (some JRE image) will only be downloaded once by service-base.

[remmeier]

Our application looks like:

• we have a Git repository with a single Gradle build building 30 (sub)projects
• those projects build around 10 different REST services (and further init-containers)
• the REST service have the same technology stack (Spring, Jackson, micrometer, crnk,…). Footprint of everything is quite substantial in terms of size.
• given same technology stack we build with Jib 10 Docker images that look very much alike. The business functionality is different for each service, but third-party libraries like Spring are to 95% identical. But those other 5% cause for new third-party layers to be created for every one of those 10 Docker images. So it would be much much more efficient we were able to put the 95% into a seperate layer from those other 5%.
• in Gradle we already have a “service-base” project where all the shared dependencies are defined.
• as the simplest solution we guessed that being able to create a base image for “service-base” and then make us of that base image in all 10 other services we can achieve exactly with what we want and gain proper docker layering.
• With a few further customization options that float around in other tickets, one could achieve this request here externally without touching Jib. But the problem seems quite generic, proper docker layering a frequent request and the result can bring large performance gains, so native support seems worthwhile to have.
• once we are done building all the images we install them with Helm (https://github.com/rmee/gradle-plugins/tree/master/helm) in a subsequent deployment step.

Gradle up-to-date checking, caching, size of build artifacts and incremental updates of Kubernetes are one of our primary concerns/goals in the area to make it efficiently even tough the project is larger in size. We do not want to split up the project into many small parts and have to start versioning and manage them independently as it would drastically increase our efforts. Over the past months I created a number of tickets to make it work. It works in general, but the little things still hurt. This one being a primary one.

[JoeWang1127]

close as not planned