HBase to Bigtable replication tool not compatible with HBase < 1.4.0

I tried to migrate an older HBase cluster (1.2.0) to Bigtable recently (trying to get rid of HBase and replace it with something currently supported+more reliable) and the HBase to Bigtable replication tool from this repo seemed like the ideal tool for the job. I followed the instructions in the guide here https://github.com/googleapis/java-bigtable-hbase/tree/main/hbase-migration-tools/bigtable-hbase-replication#hbase-bigtable-replication, but following step 7 in the migration guide (offline export/import of HBase snapshots into Bigtable), I was unable to get replication to actually start. Upon digging through the HBase RegionServer logs, I discovered replication was erroring out with: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.replication.regionserver.MetricsSource.incCounters.

After investigating things for a few hours, I discovered that the MetricsSource.incCounters() method is not available in HBase until version 1.4.0 - which makes the bigtable-hbase-1.x-replication-1.0.1-jar-with-dependencies.jar migration tool incompatible with HBase versions older than that.

Please create an implementation of IncompatibleMutationAdapter that does not perform a call to MetricsSource.incCounters() so that the HBase to Bigtable migration tool can be used with versions of HBase < 1.4.0. From my limited understanding of the documentation, it should be possible to use the new IncompatibleMutationAdapter via the parameter google.bigtable.incompatible_mutation.adapter.class so you can have a migration workflow for those poor souls such as myself trying to migrate off of HBase versions older than 1.4.0 without breaking existing functionality.

Environment details

1. Other (Bigtable->HBase replication tool version 1.0.1)
2. OS type and version: Ubuntu 16.04
3. Java version: 1.8.0_181
4. Version(s):
   • HBase: 1.2.0-cdh5.14.0 😬
   • Migration tool: bigtable-hbase-1.x-replication-1.0.1-jar-with-dependencies.jar

Steps to reproduce

1. Attempt to follow the migration guide here: https://github.com/googleapis/java-bigtable-hbase/tree/main/hbase-migration-tools/bigtable-hbase-replication#hbase-bigtable-replication with an HBase version < 1.4.0.
2. Stack trace will appear in the logs and replication will become stuck:

hbase(main):001:0> status 'replication'
version 1.2.0-cdh5.14.0
3 live servers
    hbase-server-name-1:
       SOURCE: PeerID=2, AgeOfLastShippedOp=0, SizeOfLogQueue=3, TimeStampsOfLastShippedOp=Wed Dec 31 19:00:00 EST 1969, Replication Lag=1651246442790
       SINK  : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Fri Apr 29 10:28:49 EDT 2022
    hbase-server-name-2:
       SOURCE: PeerID=2, AgeOfLastShippedOp=0, SizeOfLogQueue=3, TimeStampsOfLastShippedOp=Wed Dec 31 19:00:00 EST 1969, Replication Lag=1651246441482
       SINK  : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Fri Apr 29 10:33:03 EDT 2022
    hbase-server-name-3:
       SOURCE: PeerID=2, AgeOfLastShippedOp=0, SizeOfLogQueue=3, TimeStampsOfLastShippedOp=Wed Dec 31 19:00:00 EST 1969, Replication Lag=1651246441911
       SINK  : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Fri Apr 29 10:26:32 EDT 2022

Stack trace

2022-04-28 09:46:11,599 INFO com.google.cloud.bigtable.hbase.replication.CloudBigtableReplicator: Starting replication to CBT.
2022-04-28 09:46:11,599 ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unexpected exception in ReplicationSource
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.replication.regionserver.MetricsSource.incCounters(Ljava/lang/String;J)V
        at com.google.cloud.bigtable.hbase1_x.replication.metrics.HBaseMetricsExporter.incCounters(HBaseMetricsExporter.java:35)
        at com.google.cloud.bigtable.hbase.replication.adapters.IncompatibleMutationAdapter.<init>(IncompatibleMutationAdapter.java:98)
        at com.google.cloud.bigtable.hbase.replication.adapters.ApproximatingIncompatibleMutationAdapter.<init>(ApproximatingIncompatibleMutationAdapter.java:50)
        at com.google.cloud.bigtable.hbase.replication.adapters.IncompatibleMutationAdapterFactory.createIncompatibleMutationAdapter(IncompatibleMutationAdapterFactory.java:40)
        at com.google.cloud.bigtable.hbase.replication.CloudBigtableReplicator.start(CloudBigtableReplicator.java:201)
        at com.google.cloud.bigtable.hbase1_x.replication.HbaseToCloudBigtableReplicationEndpoint.doStart(HbaseToCloudBigtableReplicationEndpoint.java:46)
        at com.google.common.util.concurrent.AbstractService.start(AbstractService.java:90)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:251)

External references such as API reference guides

The org.apache.hadoop.hbase.replication.regionserver.MetricsSource.incCounters method first appears in this commit here (which first appears in HBase 1.4.0): https://github.com/apache/hbase/commit/cb02be38ab20ec5343fc9a7450bed33461e38f10

[vermas2012]

Great to hear that @jstaf!

Thank you for your detailed analysis, you articulated the problem very well which reduced the “time to fix” for us.

[vermas2012]

The fix is in, we will cut a release soon, that should provide a jar which handles this issue properly.

Please note that this fix disables the custom metrics. If you are issuing incompatible mutations, please check the logs. The logs will be emitted by classes in package com.google.cloud.bigtable.hbase.replication.adapters, please make sure that logger is set to WARN or above for incompatible mutations that are DROPPED. Setting it to INFO will log all the incompatible mutations.

By default, all the replication related classes are logged at INFO level. https://github.com/googleapis/java-bigtable-hbase/blob/68d4a012a158e42759facae7377ff1aa75fcf0fd/hbase-migration-tools/bigtable-hbase-replication/bigtable-hbase-replication-core/src/main/resources/log4j.properties#L18