Time to first OK request increased (doubled) for MP application between 1.3.4.Final and 1.7.2.Final

We have testing application which is using all MP specs - https://github.com/quarkus-qe/quarkus-startstop/blob/master/app-full-microprofile/pom.xml. I’m using beefed up bare metal machine with Intel® Xeon® CPU E5-2695 v2 @ 2.40GHz (24 processors reported, 12 cores) and 32 GB RAM. This bare metal machine is running on RHEL 7.8.

Time to first OK request doubled for this MP application between 1.3.4.Final and 1.7.2.Final

Times for 1.3.4.Final are ~36 ms for the first OK request. Times for 1.7.2.Final are ~71 ms for the first OK request.

There are 2 factors that can influence this diff

• different GraalVM version - 19.3 vs. 20.1
• different Quarkus version - 1.3.4.Final and 1.7.2.Final

Reproducer:

git clone https://github.com/quarkus-qe/quarkus-startstop.git
cd quarkus-startstop

alias setJdkGraalVM='sdk use java 20.1.0.r11-grl && export GRAALVM_HOME=$JAVA_HOME'
alias setJdkGraalVM19='sdk use java 19.3.1.r11-grl && export GRAALVM_HOME=$JAVA_HOME'

setJdkGraalVM
mvn clean test -Dtest=StartStopTest#fullMicroProfileNative -Ptestsuite -Dquarkus.version=1.7.2.Final
cat testsuite/target/archived-logs/io.quarkus.ts.startstop.StartStopTest/fullMicroProfileNative/measurements.csv

setJdkGraalVM19
mvn clean test -Dtest=StartStopTest#fullMicroProfileNative -Ptestsuite -Dquarkus.version=1.3.4.Final
cat testsuite/target/archived-logs/io.quarkus.ts.startstop.StartStopTest/fullMicroProfileNative/measurements.csv

You can search for AVG timeToFirstOKRequest without min and max values in console output to get the number without looking at measurements.csv.

Details from my bare metal machine:

## 1.3.4.Final
cat testsuite/target/archived-logs/io.quarkus.ts.startstop.StartStopTest/fullMicroProfileNative/measurements.csv
App , Mode , buildTimeMs , timeToFirstOKRequestMs , startedInMs , stoppedInMs , RSSKb , FDs
FULL_MICROPROFILE , NATIVE , 119838 , 41 , 37 , 4 , 93112 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 35 , 28 , 4 , 94664 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 36 , 29 , 3 , 94772 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 38 , 31 , 4 , 101112 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 36 , 30 , 3 , 97132 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 37 , 29 , 3 , 100844 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 37 , 29 , 4 , 98552 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 35 , 28 , 3 , 109104 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 36 , 30 , 3 , 94712 , 175
FULL_MICROPROFILE , NATIVE , 119838 , 35 , 28 , 3 , 94960 , 175

## 1.7.2.Final
cat testsuite/target/archived-logs/io.quarkus.ts.startstop.StartStopTest/fullMicroProfileNative/measurements.csv
App , Mode , buildTimeMs , timeToFirstOKRequestMs , startedInMs , stoppedInMs , RSSKb , FDs
FULL_MICROPROFILE , NATIVE , 121474 , 68 , 37 , 4 , 98260 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 71 , 36 , 3 , 104248 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 72 , 37 , 3 , 120580 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 69 , 36 , 4 , 110680 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 71 , 38 , 3 , 108200 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 75 , 40 , 3 , 118844 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 70 , 36 , 3 , 112772 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 72 , 37 , 3 , 112372 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 69 , 35 , 4 , 106216 , 174
FULL_MICROPROFILE , NATIVE , 121474 , 69 , 35 , 3 , 114800 , 174

[rsvoboda]

I was checking 2.1.0.Final. It has similar timeToFirstOKRequestMs like 1.7.2.Final, the app uses less memory (rssKb). Used GraalVM 20.3 for the comparison and StartStopTest#fullMicroProfileNative test.

Also tried 2.1.0.Final StartStopTest#fullMicroProfileNative test with GraalVM 21.0.0.2 and 21.2.0, similar results as with GraalVM 20.3. Checked on my local MBP, so definitely not a performance environment.

As part of the discussion around this issue Jaeger initialization was identified as the key element in the increase.

I think we can close this issue. We will need to run the round of tests in Perf lab with Quarkus 1.11.7.Final, 2.1.0.Final and “ancient” 1.7.6.Final. @mjurc, @johnaohara fyi.

[geoand]

@rsvoboda can we close this? These versions are pretty old at this point