Dependencies for using Pulsar in Storm topologies

Pulsar version: 2.2.0

I build an uber jar for a Storm toplogy containing the following dependency as per http://pulsar.apache.org/docs/latest/adaptors/PulsarStorm/

    <properties>
		<pulsar.version>2.2.0</pulsar.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.apache.pulsar</groupId>
			<artifactId>pulsar-storm</artifactId>
			<version>${pulsar.version}</version>
		</dependency>
	</dependencies>

am getting

13900 [pulsar-client-io-3-1] WARN  o.a.p.c.i.ClientCnx - [localhost/127.0.0.1:6650] Got exception NoClassDefFoundError : org/asynchttpclient/channel/KeepAliveStrategy
java.lang.NoClassDefFoundError: org/asynchttpclient/channel/KeepAliveStrategy
	at org.apache.pulsar.client.impl.ClientCnx.newConnectCommand(ClientCnx.java:171) ~[pulsar-1.0-SNAPSHOT.jar:?]
	at org.apache.pulsar.client.impl.ClientCnx.channelActive(ClientCnx.java:151) ~[pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1402) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:921) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:616) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:525) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) [pulsar-1.0-SNAPSHOT.jar:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pulsar-1.0-SNAPSHOT.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

so I added the following dependency explicitly

		<dependency>
			<groupId>org.asynchttpclient</groupId>
			<artifactId>async-http-client</artifactId>
			<version>2.1.0-alpha26</version>
		</dependency>

which seems to solve this particular problem. Shouldn’t the documentation list that dependency?

Am then getting an endless stream of errors in the logs

72085 [pulsar-timer-8-1] WARN  o.a.p.c.i.ProducerImpl - [persistent://public/crawl/in-partition-8] [crawler-1-1540895529_0] error while closing out batch -- java.lang.NoSuchMethodError: com.scurrilous.circe.checksum.Crc32cIntChecksum.computeChecksum(Lio/netty/buffer/ByteBuf;)I
72086 [pulsar-timer-9-1] WARN  o.a.p.c.i.ProducerImpl - [persistent://public/crawl/in-partition-5] [crawler-1-1540895529_1] error while closing out batch -- java.lang.NoSuchMethodError: com.scurrilous.circe.checksum.Crc32cIntChecksum.computeChecksum(Lio/netty/buffer/ByteBuf;)I
72086 [pulsar-timer-8-1] WARN  o.a.p.c.i.ProducerImpl - [persistent://public/crawl/in-partition-6] [crawler-1-1540895529_0] error while closing out batch -- java.lang.NoSuchMethodError: com.scurrilous.circe.checksum.Crc32cIntChecksum.computeChecksum(Lio/netty/buffer/ByteBuf;)I
72086 [pulsar-timer-9-1] WARN  o.a.p.c.i.ProducerImpl - [persistent://public/crawl/in-partition-1] [crawler-1-1540895529_1] error while closing out batch -- java.lang.NoSuchMethodError: com.scurrilous.circe.checksum.Crc32cIntChecksum.computeChecksum(Lio/netty/buffer/ByteBuf;)I

I checked the dependencies and that class and method seem to be there. Any idea?

See https://github.com/DigitalPebble/storm-crawler/tree/pulsar/external/pulsar for the code I wrote based on your Storm adaptor. Thanks!

[sijie]

@jnioche I think there is a problem in the pulsar-storm pom file: https://search.maven.org/artifact/org.apache.pulsar/pulsar-storm/2.2.0/jar it is mixing shaded and unshaded versions of pulsar. working on a fix

[sijie]

@jnioche thank you for reporting this. will work on a fix.