SourceRowCount example seems not working with the higher versions of bean packages?

Basically I am trying to follow the guide from

• here https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/tree/master/java/dataflow-connector-examples
• and here https://cloud.google.com/dataflow/docs/quickstarts/quickstart-java-maven

And I wanted to run the job SourceRowCount to count the bigtable row numbers.

As I used the pom.xml from the second url above, some of the packages are newer than the first url above.

Then in encountered this issue: https://github.com/googleapis/cloud-bigtable-client/issues/2150

Then I used the solution mentioned in the #2150 to change bigtable-hbase-beam to 1.5.0, the local error of com.google.bigtable.repackaged.io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. is gone.

However I still have Permission denied error on the log of dataflow job while scan bigtable. I believed that I have set the service accounts correct for my project.

Then I changed <beam.version>2.13.0</beam.version> to <beam.version>2.11.0</beam.version>, my job stopped reporting Permission denined issue on dataflow. However the job still failed after a long hour with message: Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h.

So my question now is: How can I do a bigtable row count properly. Should I just use the exact configuration of the SourceRowCount example or?

[igorbernstein2]

I’m sorry for the troubles you are having and thank you for the time you’ve spent debugging the issue. The issue is caused by an inconsistent configuration of the bigtable-batch.googleapis.com endpoint and gRPC’s JWT support. I just added a workaround: #2187 that should fix the issue until we get it sorted on the serverside.

[liufuyang]

@rahulKQL Yes, I tried again with the official code and beam version 2.13.0 and there was no issues. But if I use the code and pom.xml above I provided, I would get this issue, even with the same service account setup. However I guess I should probably just use the example code then.