Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Offload data from bk to aws-s3 error on 2.5.2

See original GitHub issue

Describe the bug

conf/standalone.conf:

 managedLedgerOffloadDriver=aws-s3
 s3ManagedLedgerOffloadBucket=test-pulsar-offload
 s3ManagedLedgerOffloadRegion=us-west-2
 s3ManagedLedgerOffloadServiceEndpoint=http://s3.amazonaws.com

the broker error as follows:

NarClassLoader@3a8dc651] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
13:36:59.190 [pulsar-ordered-OrderedExecutor-6-0] INFO  org.apache.bookkeeper.mledger.offload.OffloaderUtils - Found and loaded 1 offloaders
13:36:59.196 [pulsar-ordered-OrderedExecutor-6-0] ERROR org.apache.pulsar.broker.PulsarService - create ledgerOffloader failed for namespace public/functions
org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: Either s3ManagedLedgerOffloadRegion or s3ManagedLedgerOffloadServiceEndpoint must be set if s3 offload enabled
	at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:836) ~[org.apache.pulsar-pulsar-broker-2.5.2.jar:2.5.2]
	at org.apache.pulsar.broker.PulsarService.lambda$getManagedLedgerOffloader$4(PulsarService.java:800) ~[org.apache.pulsar-pulsar-broker-2.5.2.jar:2.5.2]
	at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877) ~[?:1.8.0_201]
	at org.apache.pulsar.broker.PulsarService.getManagedLedgerOffloader(PulsarService.java:792) ~[org.apache.pulsar-pulsar-broker-2.5.2.jar:2.5.2]
	at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$33(BrokerService.java:1065) ~[org.apache.pulsar-pulsar-broker-2.5.2.jar:2.5.2]
	at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [org.apache.pulsar-managed-ledger-2.5.2.jar:2.5.2]
	at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.apache.pulsar.broker.PulsarServerException: Either s3ManagedLedgerOffloadRegion or s3ManagedLedgerOffloadServiceEndpoint must be set if s3 offload enabled
	at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:829) ~[org.apache.pulsar-pulsar-broker-2.5.2.jar:2.5.2]
	... 10 more

To Reproduce Steps to reproduce the behavior:

connfig conf/standalone.conf
start pulsar 2.5.2 standalone

Expected behavior The broker can start normally

Issue Analytics

State:
Created 3 years ago
Comments:5 (5 by maintainers)

Top GitHub Comments

1reaction

lukestephenson-zendeskcommented, Jun 11, 2020

I’ve also encountered the same issue with pulsar 2.5.2.

From reviewing commits, it looks like it was introduced here https://github.com/apache/pulsar/commit/c3fa923e46ae37c7bab5503dbd40c3a0c54fe154#diff-bb90edad9f69a5d791cb43b04d9a4c82R1036-R1045

Note that the ManagedLedgerOffloadDriver is copied across, but not other fields on the configuration.

That code has since been removed in https://github.com/apache/pulsar/pull/7011/files#diff-bb90edad9f69a5d791cb43b04d9a4c82L1066, so fingers crossed this is fixed in pulsar 2.6.0.

0reactions

jiazhaicommented, Jun 19, 2020

since 2.6.0 is released, would like to close this issue.

Top Results From Across the Web

AWS S3 offloader - StreamNative Hub

This tiered storage solution extends the storage capabilities of Pulsar by offloading data from Apache BookKeeper to scalable cloud-native storage or files ...

Resolve errors uploading data to or downloading data from ...

I want to download data from Amazon Aurora and upload it to Amazon S3. How can I resolve an error I received while...

Use AWS S3 offloader with Pulsar

Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data...

WP Offload Media Changelog - Delicious Brains

Read all about the latest updates to WP Offload Media in our changelog. ... Improvement: Some common AWS S3 errors are caught and...

S3 Error: The difference between the request time and the ...

I have error The difference between the request time and the current time is too large when call method amazons3.ListObjects