Standalone broker on MacOS often fail to start after workstation sleep or reboot
See original GitHub issueDescribe the bug Often, but not always, if I start Pulsar standalone on my workstation and then close/reopen my laptop lid or restart the computer, subsequent attempts to start the standalone broker fail. The JVM process keeps running and emits some errors in its logs (see attachment), but never reaches a connectible state.
Repeatedly restarting the broker usually causes this condition to go away. Sometimes that doesn’t work, and I have to remove all broker data files (i.e. reinstall Pulsar) to make it start.
This only seems to happen when publishes have been recorded to a persistent topic.
This does not happen when using Pulsar Standalone in Docker, or on my Linux laptop. It appears to be MacOS specific.
To Reproduce
brew install apache-pulsar
brew services start apache-pulsar
- Verify that Pulsar is connectible on localhost:6650.
- Publish some messages to a persistent topic such that the message data and the topic itself are not automatically deleted (i.e. at least one ledger is created and persists).
- Close laptop lid or otherwise induce sleep.
- Reopen laptop and attempt to connect to Pulsar; it will sometimes fail.
brew services restart apache-pulsar
and attempt to connect to Pulsar; it will sometimes fail.
Expected behavior
- Sleep/resume should not break Pulsar. It should either be available after resumption, or the service/JVM process should fully crash so that it reports not-running if it cannot handle connection requests after resumption.
- Restarting the service should result in Pulsar either being connectible or not having a JVM process. If Pulsar Standalone can’t come up in a connectible state, its process shouldn’t hang around (if the process stays around, service managers think it’s still running and thus won’t report a failure or auto-restart it).
Desktop (please complete the following information):
- OS: MacOS monterey, intel.
- Pulsar standalone 2.9.2 and 2.9.1 both exhibit this issue.
Additional context
brew services
is likely not the problem. It doesn’t do anything special, and Pulsar does sometimes work when restarted via brew services
.
Attached is a copy of my logs from a broker start attempt that did not become connectible after 5min. nostart.log
Issue Analytics
- State:
- Created a year ago
- Comments:12 (9 by maintainers)
Top GitHub Comments
That seems to work, at least so far. Which of the
broker.conf
,proxy.conf
,standalone.conf
andbookkeeper.conf
config files should I setadvertisedAddress
in for a standalone broker?@michaeljmarshall you may still be onto something; just because the IP in my logs is an RFC1918 address doesn’t mean it necessarily resolves to my host (or a router). I’ll try setting
advertisedAddress
; if that makes the problem disappear, I can check that into the Homebrew formula for Pulsar.