Increased startup time in spring boot jar deployment vs maven startup

We are running fat jar deployment on prod & stg and maven deployment on local. We have been observing consistently that deployment time is considerably less on local with maven startup and rediculously high with fat jar deployment. (It takes approx 90 sec on local vs ~900-1000 sec with fat jar on same configuration and hardware )

While debugging the issue I found that the increased time is taken during ApplicationContext.refresh for bean creation, specifically during component scanning. With debug logs enabled for class PathMatchingResourcePatternResolver. I found that with maven deployment it directly looks for class files in target directory matching the specified pattern, whereas during jar deployment it opens jar file and embedded jar files in BOOT-INF/lib directory and then looks for class files matching the pattern.

My speculation is that the application is continuously looking for patterns in fat jar file by decompressing and searching is proving to be less efficient than searching in directory tree directly.

Example Log line in Fat jar:

[v1] [17:04:53] [] - Looking for matching resources in jar file [jar:file:/root/ofb/web/target/web-1.0.jar!/BOOT-INF/lib/core-1.0.jar]

Example log line in mvn spring-boot deployment

v1|2017/12/05 16:58:57.362||||main|DEBUG|?|Looking for matching resources in directory tree [/Users/bhuvanrawal/git/ofb/core/target/classes/com/ofb/core/invoices/adhoc/repository]

Is there a potential way to inflate/explode the library jars once just before deployment and start the application with only class files.

[bhuvanrawal]

@snicoll the application has 45244 classes when loaded in jvm. Please find the jstat -class output

Loaded  Bytes  Unloaded  Bytes     Time
 45244 75505.5        8    10.1      48.70

Here is the dependency tree of the application mvn_dependency_tree_ofb.txt

There are around 245 dependent jars when I look up in BOOT-INF/lib directory totalling 147MB in all. The jar which contains our proprietary code is 15 MB (contains the classes scanned by component scans). The application for which I have sent logs is a web service application build on spring boot web(version 1.4.4.RELEASE), with springframework core (4.3.6.RELEASE). We are using spring data elasticsearch, jpa, redis, hibernate so multiple component scans are being configured in different modules. We tried optimising the scans but it eventually boils down to why jar is being slow and mvn spring-boot is fast (we can do with 80-90 sec deployment as in case of non jar deployment), I assume same component scans must be running in both jar as well as.

On profiling the application I have clearly found that there are multiple pauses where jar version gets stuck for 2-3 mins multiple times with stack being stuck in RandomAccessFile as can be seen in main-thread-startup-stacktrace. During this time 1 core of cpu is utilised 100% by the main thread.

Do let me know if I can provide any additional stuff to help debug this.

[cawak]

Hi, we had the same issue when we tried to upgrade Spring boot to version 1.5.9 from 1.2.5 and Spring to 4.3.13 from 4.1.7. It happened when we scanned for the resources using the <import resource="classpath*:*some-context.xml" /> and as here, I saw in the log that it took much more time to look for this xml in each jar in the classpath. So what I did is to narrow the matching path: <import resource="classpath*:somePath/*some-context.xml" /> And instead of passing through dozens of jars, it found the only one I need (but again the search itself in this jar took much more than previous like in the logs https://github.com/spring-projects/spring-boot/issues/11261#issuecomment-358678558.

Well, I was very curious and played a little with the antivirus and boom. The parent control of Norton seemed to slow things. Anyway, our service now boots even faster than before (won a couple of seconds).

Now I’m not sure why it’s happening only to the new versions of Spring and Spring boot (as I understood you changed the classloading mechanism), but I guess it can save time for other people.