PIP-156: Build and Run Pulsar Server on Java 17

Motivation

Currently, Pulsar requires Java8 or Java11 to build[1] and run[2], which is quite out-dated, as the latest released Java version is 18[3], as of 4/14/2022 — the Java community observed that the recent Java versions have improved numerous features such as Garbage Collection[4.1][4.2] and Text Blocks[5] . Hence, as a regular maintenance, it has been recommended to upgrade the Pulsar Java version to the latest LTS(Long Term Support) version, 17[3] for both build and runtime.

We will keep the Pulsar Java client target version as-is, to Java8, for the client runtime compatibility — Pulsar client should be able to run on customers’ current java runtime environment.

We will update Pulsar CLIs runtime version to Java17, as CLIs are expected to run on the Admin environment - CLI tools currently require the Pulsar distribution, and therefore it can be assumed that existing users can choose to use Java17 for running CLI tools

[1] https://github.com/apache/pulsar#build-pulsar [2] https://pulsar.apache.org/docs/en/deploy-bare-metal/#requirements [3] https://www.java.com/releases/ [4.1] https://blogs.oracle.com/javamagazine/post/understanding-the-jdks-new-superfast-garbage-collectors [4.2] https://kstefanj.github.io/2021/11/24/gc-progress-8-17.html [5] https://docs.oracle.com/en/java/javase/15/text-blocks/index.html (Could be useful for Pulsar Function format)

Goal

• Upgrade the Pulsar server(broker) Java version to 17 for both build and runtime to adapt the latest Java improvements
• Keep the the pulsar Java client target version to 8 for the current client runtime environment compatibility (client-server shared modules will also stay at Java 8 release)
• Upgrade Pulsar CLIs runtime version to Java17 (CLIs are expected to run on the Admin env)
• Revisit Java runtime flags for ZK, BK and Pulsar to reflect improvement from Java 17. For example, default using ZGC(or Shenandoah GC) instead of G1GC.
• Test Pulsar performance improvement on Java 17

API Changes

• None

Implementation

High Level Change

Currently,

• Java8 or Java11 is required to build pulsar
• Java8 or Java11 is required to run pulsar-broker
• Java8(>=) is required to run pulsar-client
• Default GC for ZK, BK and Pulsar is G1GC

As a result,

• Java17 is required to build pulsar
• Java17 is required to run pulsar-broker
• Java8(>=) is required to run pulsar-client
• Default GC for ZK, BK and Pulsar is ZGC( or Shenandoah GC)

Assumptions

• We expect Pulsar dependency libraries such as BookKeeper and ZooKeeper to work well on Java17. The compatibility needs to be tested. Any blocking issues will be reported and resolved as sub tasks of this work.
• We assume that ZGC and Shenandoah GC performs better than G1GC for Pulsar. We will verify this default GC and its java flag change by the performance tests.

Related Community PR

• Migrate Docker images and CI to Java 17

Test Plan

• Pass the pulsar OSS CI tests
• Pass the integration tests in the SN Continuous Integration framework
• Add Java8 Client runtime compatibility integration tests in the CI framework
• Run OpenMessaging benchmark to compare performance improvement on Java17 with the Java default runtime flag changes

Release Plan

• Include this update in the Pulsar 2.11 release

Reject Alternatives

• None

[heesung-sn]

Hi , As a follow-up task here, I have a proposal to update pulsar server default GC configs. I tried to summarize the details in my fork PR: https://github.com/heesung-sn/pulsar/pull/1, and It would be great if I could get some early feedbacks.

[lhotari]

• Update the build to make the target(release) java version to 17, except client modules.

It’s not just client modules. There are also a few libraries that are shared between the client and the broker. The maven.compiler.release, maven.compiler.target and maven.compiler.source properties should remain 8 for these modules.

For all other modules, it’s fine to set maven.compiler.release, maven.compiler.target and maven.compiler.source to 17.

• Merge your pull Migrate Docker images and CI to Java 17 #14355

That has been merged.

• Add integration tests for the different jdk compatibility tests – I will add more connector tests to CI too.

When adding these to the CI, please expand the existing GitHub Actions workflow file pulsar-ci.yaml. .

https://github.com/apache/pulsar/blob/06ba587fb92eff81785f8d463c85aaa1095292e9/.github/workflows/pulsar-ci.yaml#L333-L339

It’s recommended to use the apachepulsar/java-test-image:latest docker image for most of the new tests since it’s fast to build.

It would be useful to refactor the existing Pulsar Java client tests in a way that the same tests cases could be used in an integration test. Otherwise, it will lead to duplication in test logic. Refactoring this could be challenging. One reason is that the current test logic uses inheritance a lot, and that’s hard to reuse across different type of tests.