Dependency issue with google-cloud-pubsub and Apache Beam packages

I get a Maven install conflict between the Google Cloud pub/sub artifact:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-pubsub</artifactId>
  <version>1.53.0</version>
</dependency>

and the Apache Beam google-cloud-dataflow-java-sdk-all artifact:

    <dependency>
      <groupId>com.google.cloud.dataflow</groupId>
      <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
      <version>2.5.0</version>
    </dependency>

google-cloud-pubsub has various sub-dependencies that rely on io.grpc:grpc-core:jar:1.16.1, and subdependencies of google-cloud-dataflow-java-sdk-all depend on various versions of io.grpc:grpc-core:jar that do not match this.

The full Maven conflict error message I get is

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I’ve documented my specific issue and attempts to fix it more fully here:

https://stackoverflow.com/questions/53424272/maven-conflict-in-java-app-with-google-cloud-core-grpc-dependency

Can someone please recommend a workaround? I need one urgently.

[sduskis]

We need to document how to use google-cloud-java libraries in Beam.

I suggest that you upgrade to beam 2.8.0 and use google-cloud-pubsub 1.49.0.

The core of the problem is that Beam and the google-cloud-library clients use different grpc versions. First, I would suggest upgrading to beam 2.8.0 which uses grpc 1.13.1. I did some digging, and found that google-cloud-pubsub 1.49.0 used grpc 1.13.1 as well.

[KishoreNamala]

@chalcrow - I encountered this error and used the workaround mentioned in the above link. It resolved the compile error but broke my pubsub client while trying to publish to a topic.

Here is the error from my pubsub client (trying to create a publisher with custom batch settings).

Exception in thread "main" java.lang.NoSuchFieldError: transportTracerFactory
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:394)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:266)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:223)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:169)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:156)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:157)
	at com.google.cloud.pubsub.v1.stub.GrpcPublisherStub.create(GrpcPublisherStub.java:161)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:149)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:78)
	at com.google.cloud.pubsub.v1.Publisher$Builder.build(Publisher.java:614)
	at com.techm.data.service.PublisherSvc.getPublisherWithCustomBatchSettings(PublisherSvc.java:155)
	at com.techm.data.service.PublisherSvc.publishMessageBatchToATopic(PublisherSvc.java:116)
	at com.techm.data.client.CreateTopicAndPublishMessages.publishObjectMessages(CreateTopicAndPublishMessages.java:99)
	at com.techm.data.client.CreateTopicAndPublishMessages.main(CreateTopicAndPublishMessages.java:158)

And here is the method creating the publisher

	public Publisher getPublisherWithCustomBatchSettings(ProjectTopicName topicName) throws Exception {
		// [START pubsub_publisher_batch_settings]
		// Batch settings control how the publisher batches messages
		long requestBytesThreshold = 5000L; // default : 1 byte
		long messageCountBatchSize = 10L; // default : 1 message

		Duration publishDelayThreshold = Duration.ofMillis(100); // default : 1 ms

		// Publish request get triggered based on request size, messages count & time
		// since last publish
		BatchingSettings batchingSettings = BatchingSettings.newBuilder()
				.setElementCountThreshold(messageCountBatchSize).setRequestByteThreshold(requestBytesThreshold)
				.setDelayThreshold(publishDelayThreshold).build();

		Publisher publisher = Publisher.newBuilder(topicName).setBatchingSettings(batchingSettings).build();
		// [END pubsub_publisher_batch_settings]
		return publisher;
	}