Split dependencies into two layers(silent and mutable)

Now Jib separates your application into multiple layers: dependencies, resources and classes, but still a question here. dependencies layer is still very big. We can split dependencies layer. Why create a new layer for dependencies? I did some research about dependencies in Java application. Some jars are very big, for example groovy jar, hibernate jar etc. These jars are very big but stable, and developers don’t change their versions frequently. The developers change the company internal artifacts regularly, or changed by SNAPSHOT automatically. For example, a Spring Boot Application is almost 30+M, if we create a layer with following artifacts included, the original dependencies layer will be 5M almost, an new 25M layer(silent dependencies layer) could be stable for a long time. if the developer changes internal artifact version, only push the 5M layer(mutable dependencies layer), not original 30M layer again.

In Java, another consideration is SNAPSHOT. During development and testing phases, most guys use SNAPSHOT version. These jars will be changed regularly from continue integration system. For some case, for example, security jars always were delivered as SNAPSHOT version. 100K SNAPSHOT jar will make you rebuild the dependencies layer.

We can introduce a silent dependencies layer and group stable & big jars according to groupId and artifact wild match, or mutable dependencies layer. jib-maven-plugin Configuration:

<silentDependencies>org.springframework:*, org.hibernate:*, *:commons-*, org.webjars:*</silentDependencies>
<mutableDependencies>com.yourcompany:*,*:*:*-SNAPSHOT</mutableDependencies>

the Dockerfile will copy slient jars into a new layer

COPY silent-libs /app/libs/
COPY mutable-libs /app/libs

all artifacts should be in libs directory to make dependencies check easy.

The followings are some very popular artifacts size information for references, and of course the developers can add other jars. for a spring boot application, almost silent dependencies layer is almost 80+% in size, and mutable dependencies layer almost 20-% in size.

Spring Framework related

• spring all: spring & spring boot & spring cloud = 12M
• reactor all: 3M

Apache

• apache commons all: 1.8M

JVM languages

• groovy: 4.7M
• jruby: 10M
• kotlin all: 3.5M
• scala: 5.7M

web server

• tomcat all: 3.6M
• jetty all: 2.4
• undertow all: 3M

Java EE

• javax.*: stable, size is small
• javaee: 2M

Driver & client

• Oracle: ojdbc8.jar 4M
• MySQL: 2M
• H2 Database: 2M
• postgresql: 0.7M
• sqlserver: 0.9M
• kafka: 7.5M

Misc

• hibernate: 7.6M
• jackson all: 2M
• webjars: font-awesome(7.6M）, bootstrap(1.0M), jquery(1.5M)
• byte-buddy: 2.9M
• aspectj: aspectj* 3M
• snappy-java: 1.1M
• netty all: 2.3M
• freemarker: 1.5 M
• thymeleaf: 1.0 M
• bouncycastle: 4M

[chanseokoh]

FYI @velo @liqweed @linux-china @ivan-gammel @stigkj @d5nguyenvan @Mart-Bogdan @Ameausoone @steven-sheehy @mdiskin @yamass

The Jib Extension Framework is now available with the latest Jib versions. You can easily extend and tailor the Jib plugins behavior to your liking.

We’ve written a general-purpose layer-filter extension that enables fine-grained layer control, including deleting files and moving files into new layers.

• Maven Layer-Filter Extension
• Gradle Layer-Filter Extension

For general information about using and writing extensions, take a look at the Jib Extensions repo.

[linux-china]

@coollog I added consideration for SNAPSHOT version in Java, and these jars should be considered as mutable.