Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

failed to get Partitioned metadata : Policies not found for mytenant/mynamespace namespace

See original GitHub issue

Describe the bug We have 4 pulsar cluster running on kubernetes with geo-replication. All worked well. Before going to prod, we did some load testing. And during one of them (last one at time of writing), pulsar brokers started to loop on exceptions like:

13:30:24.053 [pulsar-io-21-3] WARN  org.apache.pulsar.client.impl.BinaryProtoLookupService - [non-persistent://mytenant/mynamespace/mytopic-98f088294f0c7a509ab4b1a5412b79308a75d50e] failed to get Partitioned metadata : Policies not found for session/user_events namespace
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Policies not found for mytenant/mynamespace

Note : we have a lot of non-persistent / not partitioned topics in that namespace (23k). Too many?

To Reproduce Well, difficult. We got this “by chance” during a load test and I cannot find out what caused it

During my investigation so far I tried many things, including:

pulsar-admin namespaces policies mytenant/mynamespace

that returns information without any problem. But I really want to understand what happened and not only restart the cluster and start from fresh.

Everything looks fine (according to logs):

  • zookeeper
  • global zookeeper (used for geo replication)
  • bookkeeper (I checked even if we do not persist those topics)

Only brokers in all regions do log a lot (well, maybe not that much per topic, but with so many topics it is quite a lot).

Any clue on how to investigate / fix it (in case it happens in prod once we do that move) ?

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:14 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
f-ldcommented, Oct 22, 2019

Getting the issue again with all kind of topics :

  • non partitioned global topics
  • partitioned global topics
  • local (i.e. only one cluster) non partitioned topics
  • local (i.e. only one cluster) partitioned topics

And for the latter, I checked that all 3 zk instances have the same content for /namespace/<mytenant>/<mynamespace>

When I do a pulsar-admin topics list tenant/namespace I get a response HTTP 500 Internal Server Error.

Checking the logs I can see on one broker (the one handling the request):

17:25:53.474 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.1.122 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/persistent/tenant/namespace HTTP/1.1" 200 675 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.516 [pulsar-web-30-1] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x15555555_0x2aaaaaaa
17:25:53.516 [pulsar-web-30-4] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x55555554_0x6aaaaaa9
17:25:53.516 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x15555555_0x2aaaaaaa HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.516 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x55555554_0x6aaaaaa9 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.516 [pulsar-web-30-7] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x95555553_0xaaaaaaa8 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.518 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xd5555552_0xeaaaaaa7 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.519 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x3fffffff_0x55555554
17:25:53.519 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x3fffffff_0x55555554 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.519 [pulsar-web-30-4] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0xbffffffd_0xd5555552
17:25:53.520 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x00000000_0x15555555 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.520 [pulsar-web-30-8] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x2aaaaaaa_0x3fffffff
17:25:53.520 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xbffffffd_0xd5555552 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 3
17:25:53.520 [pulsar-web-30-8] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x2aaaaaaa_0x3fffffff HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.521 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x6aaaaaa9_0x7ffffffe HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.521 [pulsar-web-30-7] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x7ffffffe_0x95555553
17:25:53.522 [pulsar-web-30-7] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x7ffffffe_0x95555553 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 4
17:25:53.522 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xeaaaaaa7_0xffffffff HTTP/1.1" 200 57 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.524 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0xaaaaaaa8_0xbffffffd
17:25:53.524 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xaaaaaaa8_0xbffffffd HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.529 [pulsar-web-30-5] ERROR org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Failed to get list of topics under namespace tenant/namespace
java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException$NotFoundException: Tenant does not exist
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_212]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_212]
	at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:184) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.Server.handle(Server.java:503) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException$NotFoundException: Tenant does not exist
	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:214) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.apache.pulsar.client.admin.internal.TopicsImpl$5.failed(TopicsImpl.java:159) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	... 2 more
Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found
	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1086) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:99) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	... 2 more
17:25:53.532 [pulsar-web-30-5] INFO  org.eclipse.jetty.server.RequestLog - 10.11.1.122 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace HTTP/1.1" 500 5406 "-" "Pulsar-Java-v2.4.1" 21

On other 2 brokers I get:

17:25:53.522 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x95555553_0xaaaaaaa8?authoritative=true HTTP/1.1" 404 34 "-" "Pulsar-Java-v2.4.1" 3
17:25:53.525 [pulsar-web-30-5] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x6aaaaaa9_0x7ffffffe?authoritative=true HTTP/1.1" 404 34 "-" "Pulsar-Java-v2.4.1" 2

and

17:25:53.521 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xd5555552_0xeaaaaaa7?authoritative=true HTTP/1.1" 200 113 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.524 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x00000000_0x15555555?authoritative=true HTTP/1.1" 200 170 "-" "Pulsar-Java-v2.4.1" 2

On zookeeper, on all 3 nodes, if I check the content of /namespace/tenant/namespace I get:

[zk: localhost:2181(CONNECTED) 0] ls /namespace/tenant/namespace
[0xd5555552_0xeaaaaaa7, 0x6aaaaaa9_0x7ffffffe, 0xeaaaaaa7_0xffffffff, 0x95555553_0xaaaaaaa8, 0x00000000_0x15555555]
[zk: localhost:2181(CONNECTED) 1] get /namespace/tenant/namespace/0xd5555552_0xeaaaaaa7
{"nativeUrl":"pulsar://10.11.0.111:6650","httpUrl":"http://10.11.0.111:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 2] get /namespace/tenant/namespace/0x6aaaaaa9_0x7ffffffe
{"nativeUrl":"pulsar://10.11.3.88:6650","httpUrl":"http://10.11.3.88:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 3] get /namespace/tenant/namespace/0xeaaaaaa7_0xffffffff
{"nativeUrl":"pulsar://10.11.0.50:6650","httpUrl":"http://10.11.0.50:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 4] get /namespace/tenant/namespace/0x95555553_0xaaaaaaa8
{"nativeUrl":"pulsar://10.11.3.88:6650","httpUrl":"http://10.11.3.88:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 5] get /namespace/tenant/namespace/0x00000000_0x15555555
{"nativeUrl":"pulsar://10.11.0.111:6650","httpUrl":"http://10.11.0.111:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 5] get /admin/local-policies/tenant/namespace
{"bundles":{"boundaries":["0x00000000","0x15555555","0x2aaaaaaa","0x3fffffff","0x55555554","0x6aaaaaa9","0x7ffffffe","0x95555553","0xaaaaaaa8","0xbffffffd","0xd5555552","0xeaaaaaa7","0xffffffff"],"numBundles":12}}

So I indeed have some bundles that have “disappeared” from brokers. But I got no broker crash or zookeeper issue whatsoever.

But I do not see any error in zookeeper logs apart maybe things like:

14:21:52.112 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x200d5ed81570008 type:delete cxid:0x3032c zxid:0x100008631 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000
14:21:52.415 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x200d5ed81570007 type:delete cxid:0x244de zxid:0x100008634 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000
14:22:52.112 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x200d5ed81570008 type:delete cxid:0x3034d zxid:0x100008636 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000

but those are info logs.

Still a mystery for me … But learning everytime, so now going to rebuild the cluster and check a few more things before using it to ensure it is all ok

0reactions
devinbostcommented, Jan 29, 2020

We were getting this exact error when trying to deploy a source (after adding a zookeeper cluster for just configuration, giving us a local zookeeper cluster and a configuration zookeeper cluster). I started exploring in the local Zookeeper cluster, and I found that two of our brokers had different entries. It looked like this:

[zk: 172.200.0.11:2181(CONNECTED) 25] ls /namespace/pulsar/cluster-1/host04.example.domain.com:8080
[0x00000000_0xffffffff]
[zk: 172.200.0.11:2181(CONNECTED) 26] ls /namespace/pulsar/cluster-1/host05.example.domain.com:8080
[0x00000000_0xffffffff]
[zk: 172.200.0.11:2181(CONNECTED) 27] ls /namespace/pulsar/cluster-1/host06.example.domain.com:8080
[0x00000000_0xffffffff]
[zk: 172.200.0.11:2181(CONNECTED) 28] ls /namespace/pulsar/cluster-1/host07.example.domain.com:8080
[]
[zk: 172.200.0.11:2181(CONNECTED) 29] ls /namespace/pulsar/cluster-1/host08.example.domain.com:8080
[]
[zk: 172.200.0.11:2181(CONNECTED) 30] ls /namespace/pulsar/cluster-1/host09.example.domain.com:8080
[0x00000000_0xffffffff]
[zk: 172.200.0.11:2181(CONNECTED) 31] ls /namespace/pulsar/cluster-1/host10.example.domain.com:8080
[0x00000000_0xffffffff]
[zk: 172.200.0.11:2181(CONNECTED) 32] ls /namespace/pulsar/cluster-1/host11.example.domain.com:8080
[0x00000000_0xffffffff]
[zk: 172.200.0.11:2181(CONNECTED) 33] ls /namespace/pulsar/cluster-1/host12.example.domain.com:8080
[0x00000000_0xffffffff]

So, we tried deploying the source from a broker that wasn’t host07 or host08, and it succeeded without the exception.

Read more comments on GitHub >

github_iconTop Results From Across the Web

failed to get Partitioned metadata : authorization failed
This exception occurs when your role lacks the required permission. You either need the appropriate permission (consume, produce, ...
Read more >
Manage topics - Apache Pulsar
You can get the list of partitioned topics under a given namespace in the following ways. pulsar-admin; REST API; Java. $ pulsar-admin topics ......
Read more >
Policies not found for mytenant/mynamespace namespace
xiemeilong edited a comment on issue #4718: failed to get Partitioned metadata : Policies not found for mytenant/mynamespace namespace URL: ...
Read more >
Mastering Apache Pulsar
not exist. I also want to thank the editorial and content acquisition teams at O'Reilly. They challenged me to make this book as...
Read more >
Apache Pulsar for Modern Messaging - DEV Community ‍ ‍
In Kafka one has to not only add a broker (which itself is not easy), ... The configuration policies set on a namespace...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Flink pulsar sink NotSerializableException

Next page

standalone mode cannot sync MaxMessageSize to bookkeeper