Data files in Tablet differ from in-memory data

Describe the bug I saw this error while running Randomwalk tests on the 2.1 main branch. Data files in 28;7;6 differ from in-memory data

Versions (OS, Maven, Java, and others, as appropriate):

• Affected version(s) of this project: [2.1]
• OS: [Ubuntu]

To Reproduce Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):

1. Setup Uno. I had 2 tservers running locally.
2. Ingest some data for a while using Randomwalk MultiTable test
3. See error

Tserver log activity:

2022-03-15T09:17:46,688 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING !0;~<  total = 203,913,354 ingestMemory = 22,659,626
2022-03-15T09:17:47,651 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING 28;9;8  total = 226,570,072 ingestMemory = 45,313,436
2022-03-15T09:17:50,471 [tablet.Tablet] ERROR: Data files in 28;7;6 differ from in-memory data ...
2022-03-15T09:17:50,655 [tablet.files] DEBUG: Flushed 28;7;6 created hdfs://localhost:8020/accumulo/tables/28/t-00002pt/F00004hu.rf from [memory]
2022-03-15T09:17:50,657 [tablet.files] DEBUG: Flushed !0;~< created hdfs://localhost:8020/accumulo/tables/!0/table_info/F00004hz.rf from [memory]
2022-03-15T09:17:50,662 [tablet.files] DEBUG: Compacting !0;~< on i.meta.small for SYSTEM from [F00004hz.rf, A00004hy.rf] size 20 KB
2022-03-15T09:17:50,676 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING 28;7;6  total = 272,062,261 ingestMemory = 90,808,517
2022-03-15T09:17:50,676 [memory.LargestFirstMemoryManager] DEBUG: chosenMem = 45,493,788 chosenIT = 2.31 load 45,574,937
2022-03-15T09:17:50,677 [tablet.Tablet] DEBUG: MinC initiate lock 0.00 secs
2022-03-15T09:17:51,540 [memory.LargestFirstMemoryManager] DEBUG: BEFORE compactionThreshold = 0.056 maxObserved = 339,854,955
2022-03-15T09:17:51,540 [memory.LargestFirstMemoryManager] DEBUG: AFTER compactionThreshold = 0.056
2022-03-15T09:17:51,596 [constraints.ConstraintChecker] DEBUG: Loaded constraint org.apache.accumulo.server.constraints.MetadataConstraints for !0
2022-03-15T09:17:51,611 [tablet.files] DEBUG: Flushed 28;9;8 created hdfs://localhost:8020/accumulo/tables/28/t-00002pv/F00004i0.rf from [memory]
2022-03-15T09:17:51,613 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:50726; action: authenticate;
2022-03-15T09:17:51,652 [tablet.files] DEBUG: Compacted !0;~< for SYSTEM created hdfs://localhost:8020/accumulo/tables/!0/table_info/A00004i2.rf from [F00004hz.rf, A00004hy.rf]

[milleruntime]

Yeah it seems like the new consistency check could be better. I am not sure how to do that though.

I’m not sure either, but I would start by trying to figure out where in the tserver code does it already have a record about the file being flushed.

I pinged @keith-turner to see if he had any thoughts. I probably won’t get to look at this until later in the week. I still think it should be a blocker though for 2.1. I would imagine this error happening a lot on a very active cluster. Plus the error sounds scary 😨

[milleruntime]

It could be that we are just doing the check too often. We are currently calling it in 2 places:

• Tablet.closeConsistencyCheck()
• TabletServer.run() -> some checking thread()

Edit: It seems likes the error mostly occurs during the tserver thread check. I am doing more testing with RW trying to reproduce it.