Dependency conflicts with Apache Beam Cloud Dataflow runner
See original GitHub issueHi, I’m running into the following exception when using Firestore:
java.lang.NoSuchMethodError: com.google.auth.oauth2.ServiceAccountCredentials.getProjectId()Ljava/lang/String;
at com.google.firebase.FirebaseApp.getProjectId(FirebaseApp.java:307)
at com.google.firebase.ImplFirebaseTrampolines.getProjectId(ImplFirebaseTrampolines.java:43)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:32)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:26)
at com.google.firebase.cloud.FirestoreClient$FirestoreClientService.<init>(FirestoreClient.java:81)
at com.google.firebase.cloud.FirestoreClient.getInstance(FirestoreClient.java:71)
at com.google.firebase.cloud.FirestoreClient.getFirestore(FirestoreClient.java:64)
at com.google.firebase.cloud.FirestoreClient.getFirestore(FirestoreClient.java:52)
My pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>com.intellij</groupId>
<artifactId>annotations</artifactId>
<version>12.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>24.0-jre</version>
</dependency>
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.9.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.0.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.0.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.2</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.2.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.google.firebase</groupId>
<artifactId>firebase-admin</artifactId>
<version>5.8.0</version>
</dependency>
</dependencies>
</project>
and output of mvn dependency:tree
com.canigraduate.uchicago:scraper:jar:1.0-SNAPSHOT
+- org.slf4j:slf4j-jdk14:jar:1.7.25:compile
| \- org.slf4j:slf4j-api:jar:1.7.25:compile
+- com.intellij:annotations:jar:12.0:compile
+- com.google.guava:guava:jar:24.0-jre:compile
| +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
| +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
| +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
| +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
| \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
+- com.google.auto.value:auto-value:jar:1.5:provided
+- org.assertj:assertj-core:jar:3.9.0:test
+- org.junit.jupiter:junit-jupiter-engine:jar:5.0.3:test
| +- org.apiguardian:apiguardian-api:jar:1.0.0:test
| +- org.junit.platform:junit-platform-engine:jar:1.0.3:test
| | +- org.junit.platform:junit-platform-commons:jar:1.0.3:test
| | \- org.opentest4j:opentest4j:jar:1.0.0:test
| \- org.junit.jupiter:junit-jupiter-api:jar:5.0.3:test
+- org.junit.jupiter:junit-jupiter-params:jar:5.0.3:test
+- org.hamcrest:hamcrest-all:jar:1.3:test
+- org.jsoup:jsoup:jar:1.11.2:compile
+- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
| +- com.fasterxml.jackson.core:jackson-core:jar:2.8.9:compile
| +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9:compile
| +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.9:compile
| +- org.apache.avro:avro:jar:1.8.2:compile
| | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
| | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
| | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
| | +- org.apache.commons:commons-compress:jar:1.8.1:compile
| | \- org.tukaani:xz:jar:1.5:compile
| +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
| \- joda-time:joda-time:jar:2.4:compile
+- org.apache.beam:beam-runners-direct-java:jar:2.2.0:compile
+- org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
| +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
| | +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
| | \- com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0:compile
| +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
| | +- com.google.protobuf:protobuf-java:jar:3.2.0:compile
| | +- io.grpc:grpc-core:jar:1.2.0:compile
| | | +- io.grpc:grpc-context:jar:1.2.0:compile
| | | \- com.google.instrumentation:instrumentation-api:jar:0.3.0:compile
| | +- io.grpc:grpc-protobuf:jar:1.2.0:compile
| | | +- com.google.protobuf:protobuf-java-util:jar:3.2.0:compile
| | | \- io.grpc:grpc-protobuf-lite:jar:1.2.0:compile
| | \- io.grpc:grpc-stub:jar:1.2.0:compile
| +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| | +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.2.0:compile
| | +- com.google.apis:google-api-services-bigquery:jar:v2-rev355-1.22.0:compile
| | +- com.google.api:gax-grpc:jar:0.20.0:compile
| | | +- com.google.api:gax:jar:1.3.1:compile
| | | \- org.threeten:threetenbp:jar:1.3.3:compile
| | +- com.google.cloud:google-cloud-core-grpc:jar:1.2.0:compile
| | +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
| | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18:compile
| | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18:compile
| | | \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.18:compile
| | +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
| | | +- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
| | | \- com.google.http-client:google-http-client-jackson:jar:1.20.0:compile
| | +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
| | | \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.0:compile
| | +- io.grpc:grpc-auth:jar:1.2.0:compile
| | +- io.grpc:grpc-netty:jar:1.2.0:compile
| | | +- io.netty:netty-codec-http2:jar:4.1.8.Final:compile (version selected from constraint [4.1.8.Final,4.1.8.Final])
| | | \- io.netty:netty-handler-proxy:jar:4.1.8.Final:compile
| | | \- io.netty:netty-codec-socks:jar:4.1.8.Final:compile
| | +- io.grpc:grpc-all:jar:1.2.0:compile
| | | +- io.grpc:grpc-okhttp:jar:1.2.0:compile
| | | | +- com.squareup.okhttp:okhttp:jar:2.5.0:compile
| | | | \- com.squareup.okio:okio:jar:1.6.0:compile
| | | \- io.grpc:grpc-protobuf-nano:jar:1.2.0:compile
| | | \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
| | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
| | +- com.google.cloud:google-cloud-spanner:jar:0.20.0-beta:compile
| | | +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11:compile
| | | | \- com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11:compile
| | | \- junit:junit:jar:4.12:compile
| | | \- org.hamcrest:hamcrest-core:jar:1.3:compile
| | +- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile
| | +- com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0-pre3:compile
| | | +- commons-logging:commons-logging:jar:1.2:compile
| | | +- com.google.auth:google-auth-library-appengine:jar:0.7.0:compile
| | | \- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
| | +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26:compile
| | +- com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9:compile
| | \- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
| +- com.google.api-client:google-api-client:jar:1.22.0:compile
| | \- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
| +- com.google.http-client:google-http-client:jar:1.22.0:compile
| | \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
| | +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
| | \- commons-codec:commons-codec:jar:1.3:compile
| +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
| +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev213-1.22.0:compile
| +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
| +- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
| +- com.google.auth:google-auth-library-credentials:jar:0.7.1:compile
| +- com.google.auth:google-auth-library-oauth2-http:jar:0.7.1:compile
| \- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
| +- com.google.api-client:google-api-client-java6:jar:1.20.0:compile
| +- com.google.api-client:google-api-client-jackson2:jar:1.20.0:compile
| \- com.google.oauth-client:google-oauth-client-java6:jar:1.20.0:compile
\- com.google.firebase:firebase-admin:jar:5.8.0:compile
+- com.google.api-client:google-api-client-gson:jar:1.22.0:compile
| \- com.google.http-client:google-http-client-gson:jar:1.22.0:compile
| \- com.google.code.gson:gson:jar:2.1:compile
+- com.google.api:api-common:jar:1.2.0:compile
+- com.google.cloud:google-cloud-storage:jar:1.7.0:compile
| \- com.google.cloud:google-cloud-core-http:jar:1.7.0:compile
| \- com.google.http-client:google-http-client-appengine:jar:1.22.0:compile
+- com.google.cloud:google-cloud-firestore:jar:0.25.0-beta:compile
| \- com.google.api.grpc:proto-google-cloud-firestore-v1beta1:jar:0.1.20:compile
+- org.json:json:jar:20160810:compile
+- io.netty:netty-codec-http:jar:4.1.14.Final:compile
| \- io.netty:netty-codec:jar:4.1.14.Final:compile
+- io.netty:netty-handler:jar:4.1.14.Final:compile
| \- io.netty:netty-buffer:jar:4.1.14.Final:compile
| \- io.netty:netty-common:jar:4.1.14.Final:compile
\- io.netty:netty-transport:jar:4.1.14.Final:compile
\- io.netty:netty-resolver:jar:4.1.14.Final:compile
It looks like this is due to Beam pulling in old dependencies that aren’t sufficient for firebase-admin
, so I tried pinning
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
<version>0.9.0</version>
</dependency>
which resulted in #2496:
java.lang.NoClassDefFoundError: com/google/api/gax/rpc/ClientSettings
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
so I tried adding
<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax-grpc</artifactId>
<version>1.19.0</version>
</dependency>
but this results in
java.lang.NoClassDefFoundError: com/google/api/gax/rpc/TransportProvider
at com.google.cloud.firestore.FirestoreOptions.<clinit>(FirestoreOptions.java:41)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:37)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:26)
at com.google.firebase.cloud.FirestoreClient$FirestoreClientService.<init>(FirestoreClient.java:81)
at com.google.firebase.cloud.FirestoreClient.getInstance(FirestoreClient.java:71)
at which point I’m sort of stuck, since I thought this was provided by the gax-grpc
artifact. I’ve also tried putting firebase-admin
before all the beam dependencies, however this results in similar version conflicts…
Is there an easier way to resolve the dependency conflict between Beam and firebase-admin?
Issue Analytics
- State:
- Created 6 years ago
- Reactions:2
- Comments:21 (7 by maintainers)
Top Results From Across the Web
SDK and Worker Dependencies - Dataflow - Google Cloud
The Apache Beam SDKs and Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result ...
Read more >Dataflow and Bigtable dependency conflict - Stack Overflow
I assumed that the issue was coming from the two dependencies using different versions of the beam sdk, so I changed the beam-runners-google- ......
Read more >[#BEAM-6118] Maven dependency conflicts between google ...
Maven dependency conflicts between google-cloud-pubsub and Apache Beam packages. Status: Assignee: Priority: Resolution: Open. Unassigned.
Read more >Installing Python Dependencies in Dataflow - Medium
When you run a Dataflow pipeline, your pipeline may need python packages other than apache-beam. The dependency may be public packages from PyPI...
Read more >https://raw.githubusercontent.com/apache/beam/v2.2...
By default we exclude any class underneath the org.apache.beam namespace. ... Dataflow, Spark, Flink, FlinkLocal, Apex] String runner // gcpProject sets the ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hi, @kevmo314 , I have carefully reviewed your issue and spend some energy to analyze your dependency tree. For the first issue “java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: com.google.auth.oauth2.ServiceAccountCredentials.getProjectId()”, it caused by the duplicate library “com.google.auth:google-auth-library-oauth2-http”. Your project introduced three versions of this library, i.e., 0.7.1, 0.7.0, and 0.8.0. Of them, 0.7.1 is the acturally loaded one and 0.8.0 is the correct version really required by your code.
When you tried to solve the problem at first time, a new version 0.9.0 was forcedly introduced, which exacerbated the problem. So maybe the right way to work around this problem is that adding the following patch in your original pom file:
For the second problem “java.lang.NoClassDefFoundError: com/google/api/gax/rpc/ClientSettings:”, You directly introduced com.google.api:gax 1.19.0 (pretty new for your project), however, the older version exists in the dependency tree, as a result, maybe com.google.api:gax 1.9.0 is a better choice. ^_^
You can add the following patch in your pom.xml:
Then the problem may be disappeared. Hope my answer can help you.
For posterity, if I put
firebase-admin
first, then I end up with a conflict onio.grpc:grpc-core
as Beam seems to require 1.5.0. Pinning that to 1.9.0 as requested byfirebase-admin
results in