[Bug] `OutOfDirectMemoryError` when pulsar standalone start with multiple bookie

Search before asking

• I searched in the issues and found nothing similar.

Version

The master version of Pulsar.

Minimal reproduce step

1. rm data to clean the old standalone data folder.

2. Start Pulsar standalone with multiple bookies.

bin/pulsar standalone --num-bookies 2

What did you expect to see?

Start normally.

What did you see instead?

2022-10-26T16:19:58,249+0800 [BookieJournal-57152] INFO  org.apache.bookkeeper.bookie.JournalChannel - Opening journal data/standalone/bookkeeper/current/184136070a9.txn
2022-10-26T16:19:58,263+0800 [BookieStateManagerService-0] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /ledgers/available/127.0.0.1:57152
2022-10-26T16:19:58,264+0800 [main] INFO  org.apache.bookkeeper.bookie.UncleanShutdownDetectionImpl - Created dirty file in ledger dir: /Users/labuladong/IdeaProjects/pulsar/distribution/server/target/apache-pulsar-2.11.0-SNAPSHOT/data/standalone/bookkeeper/current
2022-10-26T16:19:58,265+0800 [ForkJoinPool.commonPool-worker-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 127.0.0.1:57152 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=57152, host=127.0.0.1, protocol=bookie-rpc, auth=[], extensions=[]}]}
2022-10-26T16:19:58,517+0800 [main] INFO  org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server.
2022-10-26T16:19:58,517+0800 [main] INFO  org.apache.pulsar.metadata.bookkeeper.BKCluster - New bookie '127.0.0.1:57152' has been created.
2022-10-26T16:19:58,530+0800 [main] INFO  org.apache.pulsar.metadata.bookkeeper.BKCluster - Starting new bookie on port: 57154
2022-10-26T16:19:58,530+0800 [main] INFO  org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService
2022-10-26T16:19:58,532+0800 [main] INFO  org.apache.bookkeeper.bookie.LegacyCookieValidation - Stamping new cookies on all dirs [data/standalone/bookkeeper/1/current]
2022-10-26T16:19:58,537+0800 [main] INFO  org.apache.bookkeeper.bookie.BookieResources - Using ledger storage: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
2022-10-26T16:19:58,537+0800 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Started Db Ledger Storage
2022-10-26T16:19:58,537+0800 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Number of directories: 1
2022-10-26T16:19:58,537+0800 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Write cache size: 2048 MB
2022-10-26T16:19:58,537+0800 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Read Cache: 2048 MB
2022-10-26T16:19:58,537+0800 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Creating single directory db ledger storage on data/standalone/bookkeeper/1/current
2022-10-26T16:19:58,538+0800 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1073741824 byte(s) of direct memory (used: 7516258304, max: 8589934592)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:806) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:735) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:186) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:181) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:91) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.buffer.Unpooled.directBuffer(Unpooled.java:148) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
	at org.apache.bookkeeper.bookie.storage.ldb.ReadCache.<init>(ReadCache.java:79) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.bookkeeper.bookie.storage.ldb.ReadCache.<init>(ReadCache.java:67) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:174) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:196) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.bookkeeper.bookie.BookieResources.createLedgerStorage(BookieResources.java:110) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:423) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
	at org.apache.pulsar.metadata.bookkeeper.BKCluster.startBookie(BKCluster.java:299) ~[org.apache.pulsar-pulsar-metadata-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.metadata.bookkeeper.BKCluster.startNewBookie(BKCluster.java:283) ~[org.apache.pulsar-pulsar-metadata-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.metadata.bookkeeper.BKCluster.startBKCluster(BKCluster.java:184) ~[org.apache.pulsar-pulsar-metadata-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.metadata.bookkeeper.BKCluster.<init>(BKCluster.java:137) ~[org.apache.pulsar-pulsar-metadata-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.metadata.bookkeeper.BKCluster$BKClusterConf.build(BKCluster.java:114) ~[org.apache.pulsar-pulsar-metadata-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.PulsarStandalone.startBookieWithMetadataStore(PulsarStandalone.java:454) ~[org.apache.pulsar-pulsar-broker-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:295) ~[org.apache.pulsar-pulsar-broker-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:141) ~[org.apache.pulsar-pulsar-broker-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]

Anything else?

Increasing XX:MaxDirectMemorySize in pulsar_env.sh can’t resolve this bug. No matter how much memory I give, bookie crashes if --num-bookies is greater than 1. Seems it’s a bug about cache allocator.

Are you willing to submit a PR?

• I’m willing to submit a PR!

[labuladong]

I change them to 512, then this issue fixes. Thanks! @codelipenghui @gaozhangmin

[gaozhangmin]

@labuladong , yes , just set them smaller. empty means using default value.