for flat classpath should we try put dependencies in a separate directory by default?

I saw this blog recently which was discussing why fat jars can be suboptimal: http://product.hubspot.com/blog/the-fault-in-our-jars-why-we-stopped-building-fat-jars

each build requires re-downloading all jars again; even when no dependencies change.

I wonder if there’s a way to promote base image sharing with flat classpath apps? e.g. if folks are doing CI/CD and doing lots of builds per day of the same microservice; most times its only the microservices’s code changing; the dependencies will rarely change. So it’d be awesome if one docker layer was the dependencies; then another docker layer was the apps jar. Then this would promote docker caching.

I wonder if we could implement this by default by using a dependencies folder? e.g. for microservice foo we create an install like this…

/app
  foo.jar
  dependencies/
     jackson-core-2.7.4.jar
     ...

then the Dockerfile contains 2 ADD statements; one for the dependencies, one for the builds jar.

Then hopefully docker will try cache the dependency layer as much as possible; which will help speed up docker builds, pushes, pulls

[jstrachan]

how about if by default we always ADD the dependencies first in the generated Dockerfile; then we ADD the ‘skinny jar / war’ second? (If it makes things easier lets use 2 different folders for dependencies and the ‘skinny jar/war’)

Then when we build in interactive developer mode we calculate a hash/string of all the dependencies and versions; build the image as normal - then we figure out the docker layer (docker image ID) created by adding the dependencies. (This is the trickiest bit really).

Then when we do a second build, we see the hash/string and docker layer - see that dependencies haven’t changed; so the Dockerfile we generate ignores everything from the ‘ADD dependencies’ upwards and starts FROM the previous dependencies image and just does the ADD of the skinny jar/war. So we effectively split the generated Dockerfile logically into 2 parts; and only rebuild the first part if the dependencies change.

That way we don’t have to confuse the developer into creating 2 logical docker images for each build; we just reuse the docker layers. But it lets us optimise away having to jar up 50Mb of dependencies and POST them to docker just for docker to tell us its cacheable - when we know its cached as we previously built it 😉

[jstrachan]

@chirino yeah it should totally work with docker or S2I binary builds; since its just an option to generate a smaller incremental Dockerfile that just uses the first clean build’s image output and then just copy the current project’s artefact