Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pulsar functions worker fails to initialize

See original GitHub issue

Describe the bug The Pulsar functions worker fails to initialize. This breaks Pulsar Functions functionality.

19:41:53.512 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.ArrayIndexOutOfBoundsException: -2
    at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:60) ~[com.google.guava-guava-21.0.jar:?]
    at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.sendNextRead(PendingReadOp.java:400) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.read(PendingReadOp.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.client.PendingReadOp.initiate(PendingReadOp.java:526) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.client.LedgerRecoveryOp.doRecoveryRead(LedgerRecoveryOp.java:148) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.client.LedgerRecoveryOp.access$000(LedgerRecoveryOp.java:37) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.client.LedgerRecoveryOp$1.readLastConfirmedDataComplete(LedgerRecoveryOp.java:109) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:135) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion$1.readEntryComplete(PerChannelBookieClient.java:1797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleReadResponse(PerChannelBookieClient.java:1878) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleV2Response(PerChannelBookieClient.java:1831) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1321) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
    at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.0.jar:4.9.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

**root@pulsar-broker-56d59cf97d-zwtxf:/pulsar# bin/pulsar-admin functions list
Function worker service is not done initializing. Please try again in a little while.

Reason: HTTP 503 Service Unavailable

Note: this issue persisted for a time period on the scale of hours until I gave up and tore down the cluster.

To Reproduce Reproduction steps unclear. Could possibly be related to updating existing functions.

Expected behavior Expect Pulsar functions to behave as expected - pulsar-admin functions list should not say there’s an HTTP 503, etc.

Platform (please complete the following information):

Using the pulsar-all image tagged 2.3.0.

Issue Analytics

State:
Created 5 years ago
Reactions:1
Comments:17 (11 by maintainers)

Top GitHub Comments

3reactions

sijiecommented, Jun 13, 2019

The problem was due to the StickyRead feature introduced in bookkeeper since 4.9.0. Pulsar 2.3.0 turns on this feature by default.

I am fixing this issue. At the same time, for people who encountered this issue, you can disable sticky read on broker to get around it.

bookkeeperEnableStickyReads=false

1reaction

sijiecommented, Nov 25, 2019

Closed this issue since the bookkeeper version is updated to 4.10 including the bookkeeper client fix.

Top Results From Across the Web

Deploy and manage functions worker - Apache Pulsar

Pulsar functions-worker is a logic component to run Pulsar Functions in ... that it has been initialized by bin/pulsar initialize-cluster-metadata command.

[GitHub] grantwwu opened a new issue #3715: Pulsar functions ...

grantwwu opened a new issue #3715: Pulsar functions worker fails to initalize URL: https://github.com/apache/pulsar/issues/3715 **Describe the bug** The ...

org.apache.pulsar.functions.worker.WorkerService.java ...

Here is the source code for org.apache.pulsar.functions.worker. ... dlogUri).build(); } catch (Exception e) { log.error("Failed to initialize dlog namespace ...

Apache pulsar 2.8.0 standalone failing - Stack Overflow

To start pulsar i need delete all data folder, after that it starts. 10:56:47.858 [main] ERROR org.apache.pulsar.functions.worker.

Deploying the Pulsar Inbound Connector

Set up your Pulsar Functions workers in any way that meets your ... Create the aerospike-pulsar-inbound.yml configuration file for the ...