Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

splitting bundle error

See original GitHub issue

Met the split bundle error when supply huge data into pulsar,block the data transfer

20:11:38.774 [pulsar-modular-load-manager-29-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Load-manager splitting bundle crossjh2wx/jh2wx/0x8a000000_0x8b000000 and unloading true
20:11:38.779 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.impl.NamespacesBase - [null] Split namespace bundle crossjh2wx/jh2wx/0x8a000000_0x8b000000
20:11:38.779 [pulsar-web-30-6] ERROR org.apache.pulsar.broker.web.PulsarWebResource - [null] Failed to validate namespace bundle crossjh2wx/jh2wx/0x8a000000_0x8b000000
java.lang.IllegalArgumentException: Invalid upper boundary for bundle
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) ~[com.google.guava-guava-21.0.jar:?]
        at org.apache.pulsar.common.naming.NamespaceBundles.validateBundle(NamespaceBundles.java:114) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.broker.web.PulsarWebResource.validateNamespaceBundleRange(PulsarWebResource.java:465) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.broker.web.PulsarWebResource.validateNamespaceBundleOwnership(PulsarWebResource.java:491) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.broker.admin.impl.NamespacesBase.internalSplitNamespaceBundle(NamespacesBase.java:741) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.broker.admin.v2.Namespaces.splitNamespaceBundle(Namespaces.java:330) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:183) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.Server.handle(Server.java:503) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
20:11:38.783 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 172.18.17.1 - - [30/Oct/2019:20:11:38 +0800] "PUT /admin/v2/namespaces/crossjh2wx/jh2wx/0x8a000000_0x8b000000/split?unload=true HTTP/1.1" 500 5943 "-" "Pulsar-Java-v2.4.1" 6
20:11:38.784 [pulsar-modular-load-manager-29-1] ERROR org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Failed to split namespace bundle crossjh2wx/jh2wx/0x8a000000_0x8b000000
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Internal Server Error
        at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:204) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
        at org.apache.pulsar.client.admin.internal.NamespacesImpl.splitNamespaceBundle(NamespacesImpl.java:516) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
        at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.checkNamespaceBundleSplit(ModularLoadManagerImpl.java:664) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateAll(ModularLoadManagerImpl.java:449) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_111]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_111]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error
        at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1098) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:883) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:229) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:456) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:340) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
        at org.apache.pulsar.client.admin.internal.NamespacesImpl.splitNamespaceBundle(NamespacesImpl.java:514) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
        ... 10 more

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:15 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
frankjkellycommented, Mar 15, 2022

Thanks @lhotari I don’t think that was the issue in our case but it’s certainly good to see.

1reaction
bperetocommented, Jan 28, 2022

we experience this also in version 2.8.1 - on high load (publish rate at 220k msg/s) bundle splitting seems not to be useful at this stage, because we have 1 topic (2 partitions).

setup:

  • 1 proxy
  • 2 brokers
  • 3 bookies
  • 3 zookeepers

proxy

13:58:58.588 [pulsar-proxy-io-2-1] ERROR org.apache.pulsar.client.impl.ClientCnx - [id: 0xcab5b0cc .. Close connection because received internal-server error java.lang.IllegalStateException: Namespace bundle public/default/0x00e60000_0x00e68000 is being unloaded
13:58:58.590 [pulsar-proxy-io-2-1] WARN  org.apache.pulsar.proxy.server.LookupProxyHandler - Failed to lookup topic netflow.goflow-proxy.pb-partition-1: java.lang.IllegalStateException: Namespace bundle public/default/0x00e60000_0x00e68000 is being unloaded

broker scraped logs:

13:57:59.179 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [lake03:3181, lake01:3181] is not adhering to Placement Policy. quarantinedBookies: []

13:58:03.348 [pulsar-modular-load-manager-33-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Load-manager splitting bundle public/default/0x00e60000_0x00e80000 and unloading true
13:58:03.353 [pulsar-web-41-2] INFO  org.apache.pulsar.broker.admin.impl.NamespacesBase - [null] Split namespace bundle public/default/0x00e60000_0x00e80000
13:58:03.358 [pulsar-web-41-2] ERROR org.apache.pulsar.broker.web.PulsarWebResource - [null] Invalid bundle range public/default/0x00e60000_0x00e80000, Invalid upper boundary for bundle
13:58:03.361 [pulsar-modular-load-manager-33-1] ERROR org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Failed to split namespace bundle public/default/0x00e60000_0x00e80000
org.apache.pulsar.client.admin.PulsarAdminException$PreconditionFailedException: Invalid upper boundary for bundle
13:58:58.476 [pulsar-ordered-OrderedExecutor-5-0] INFO  org.apache.pulsar.broker.namespace.OwnedBundle - Disabling ownership: public/default/0x00e68000_0x00e70000
13:58:58.478 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L396 E6613
13:58:58.587 [pulsar-io-4-8] INFO  org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to lookup null for topic persistent://public/default/netflow.goflow-proxy.pb-partition-1 with error Namespace bundle public/default/0x00e60000_0x00e68000 is being unloaded

with the workaround to disable bundle splitting loadBalancerAutoBundleSplitEnabled=false it seems stable.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to solve missing chunk code splitting errors after deploy
First of all let's understand the problem. When you enable code splitting in your bundle you are generating code chunks. These are usually...
Read more >
Bundle Splitting - SurviveJS
The split allows the client to download only the application bundle if there are changes only in the application code. The same goes...
Read more >
5 Techniques for Bundle Splitting and Lazy Loading in React
Bundling Javascript code into several files is known as bundle splitting. It enables only a tiny amount of code to be downloaded with...
Read more >
Code Splitting - webpack
This feature allows you to split your code into various bundles which can then be loaded on demand or in parallel. It can...
Read more >
JS Bundle Splitting | Docs | Fusion.js Engineering
Bundle splitting is the process of separating Javascript code into multiple files. This results in a smaller amount of code being downloaded on...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

python functions cannot find python3 interpreter

Next page

Error open RocksDB database when 'Set up a standalone Pulsar in Docker'