Excessive CPU use in idle mode

The Pulsar 2.9.1 Docker container is using ~10…20% CPU in idle mode.

  81.0% - 2,384 s io.netty.util.concurrent.FastThreadLocalRunnable.run
  80.7% - 2,376 s io.netty.util.internal.ThreadExecutorMap$2.run
  80.7% - 2,376 s io.netty.util.concurrent.SingleThreadEventExecutor$4.run
  80.7% - 2,376 s io.netty.channel.epoll.EpollEventLoop.run
  47.5% - 1,398 s io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange
  47.5% - 1,398 s io.netty.channel.epoll.Native.epollWait
  47.5% - 1,398 s io.netty.channel.epoll.Native.epollWait
  47.5% - 1,398 s io.netty.channel.epoll.Native.epollWait
  33.2% - 978 s io.netty.channel.epoll.EpollEventLoop.epollWait
  33.2% - 978 s io.netty.channel.epoll.Native.epollWait
  33.2% - 978 s io.netty.channel.epoll.Native.epollWait0
  0.0% - 17,667 µs io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks
  0.0% - 9,033 µs io.netty.channel.nio.NioEventLoop.run
  0.2% - 6,377 ms java.util.concurrent.ThreadPoolExecutor$Worker.run
  0.0% - 1,118 ms io.netty.util.HashedWheelTimer$Worker.run
  16.6% - 489 s io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run
  16.6% - 489 s io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run
  16.6% - 489 s io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run
  16.6% - 489 s io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run
  9.5% - 279 s io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait
  7.1% - 209 s io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange

To Reproduce Steps to reproduce the behavior:

1. docker run -d -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:2.9.1 bin/pulsar standalone
2. Add a postgres source: pulsar-admin source localrun --source-config-file pg.yaml (not sure if this is relevant)
   pg.yaml:

   tenant: "public"
   namespace: "default"
   name: "pg-source"
   topicName: "pg-topic"
   archive: "/mnt/pulsar-io-debezium-postgres-2.9.1.nar"
   parallelism: 1
   configs:
     plugin.name: "pgoutput"
     database.hostname: "postgres"
     database.port: "5432"
     database.user: "postgres"
     database.password: "postgres"
     database.dbname: "postgres"
     database.server.name: "postgres"
     schema.whitelist: "public"
     database.history.pulsar.service.url: "pulsar://localhost:6650"
   

3. Wait a couple of minutes
4. Notice 10…20% cpu use in the java process inside the container.
   Running top inside the container:

     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
       1 root      20   0   10.5g   1.1g  34452 S  14.0   9.1   3:27.74 java
     711 root      20   0    4244   3528   2988 S   0.0   0.0   0:00.04 bash
     724 root      20   0    6124   3288   2760 R   0.0   0.0   0:00.00 top
   

Desktop (please complete the following information):

• OS: Windows 10 x64, WSL2 Linux x64 running Docker

Logs: Couldn’t see anything wrong in the logs. See attached: pulsar.log

[rustyx]

Did some tests, not much improvement from managedLedgerCacheEvictionFrequency=1000 and PULSAR_GC=-XX:+UseG1GC. --no-functions-worker --no-stream-storage together bring ~15% CPU usage down to ~4%. Around 2/3 of that improvement seems to come from --no-stream-storage alone. Still, it is not ideal that an idle server uses 4% CPU, it drains the battery…

[lhotari]

This might be related to https://twitter.com/normanmaurer/status/1499778839563718664

Much ❤️ for people who include flamegraphs in issues when reporting performance problems in #netty. This one was a nice one https://github.com/netty/netty/issues/11695 … And here is the proposed fix https://github.com/netty/netty/pull/12145

The fix is scheduled for Netty 4.1.76.Final release.