Table.exists hangs

I’ve a Bigtable client that connects to a local emulator. Lately it has started hanging on table.exists - the only thing that was updated recently was the emulator itself via apt. A thread dump reveals (client class names snipped at the bottom)

"main" #1 prio=5 os_prio=0 tid=0x00007ff678010000 nid=0x67c2 waiting on condition [0x00007ff67f143000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007441b7ad0> (a com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation$GrpcFuture)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:469)
	at com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:293)
	at com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.readFlatRowsList(BigtableDataGrpcClient.java:327)
	at com.google.cloud.bigtable.hbase.BigtableTable.getResults(BigtableTable.java:257)
	at com.google.cloud.bigtable.hbase.BigtableTable.exists(BigtableTable.java:166)
..........

Subsequent thread dumps taken at 30 second intervals reveals the the thread is stuck at the same place.

After about 12 minutes, I get the following error -

java.io.IOException: Failed to perform operation. Operation='checkAndMutate', projectId='emulator', tableName='mquery', rowKey='149de7d2-272b-4524-ace3-8d977da6faf6#27c893ee-b50c-4b47-b08f-f3824291a62a'
	at com.google.cloud.bigtable.hbase.BigtableTable.logAndCreateIOException(BigtableTable.java:522)
	at com.google.cloud.bigtable.hbase.BigtableTable.checkAndMutate(BigtableTable.java:470)
----snipped----
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 359999872003ns
	at io.grpc.Status.asRuntimeException(Status.java:517)
	at com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:300)
	at com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.checkAndMutateRow(BigtableDataGrpcClient.java:267)
	at com.google.cloud.bigtable.hbase.BigtableTable.checkAndMutate(BigtableTable.java:467)
	... 32 common frames omitted

This happens for each call. The 12-minute interval occurs in each case. It’s brought my development to a grinding halt since I cannot use HBase/Bigtable at all. It was temporarily resolved by switching back to an older version of the emulator (gcloud 175) but the problem is back again today.

Library versions: google-cloud-core 1.8.0, google-cloud-core-grpc 1.8.0, google-cloud-core 1.8.0, bigtable-hbase-1.x 1.0.0-pre3

gcloud version output: Google Cloud SDK 177.0.0 alpha 2017.10.20 beta 2017.10.20 bigtable bq 2.0.27 cbt core 2017.10.20 gsutil 4.28 pubsub-emulator 2017.10.20

OS: Ubuntu 17.10

java -version output: java version “1.8.0_102” Java™ SE Runtime Environment (build 1.8.0_102-b14) Java HotSpot™ 64-Bit Server VM (build 25.102-b14, mixed mode)

[talonx]

@sduskis Thanks. Logged - https://github.com/GoogleCloudPlatform/google-cloud-go/issues/805

[mbrukman]

Closing this issue as it is related to the emulator, so the bug opened in the google-cloud-go repo is the correct one we should pursue, and we don’t need this one any further.