[BUG] Invalid Index on 0 (corrupted datafile?)

Version LiteDB: Datafile was created with 5.0.8. Updating to 5.0.9 or master doesn’t solve that either (but still using the datafile created with 5.0.8) OS:

• Azure: Alpine 3.12, following docker image: mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine
• Locally: Windows 10 - 2004 (Build 19041.508)

.NET: .NET Core 3.1

Describe the bug We are running a ASP.NET Core 3.1 service in a Docker container inside Azure (using Azure Container Instances). Today we got several errors while inserting an entity into a LiteDB datafile which I can’t really explain. For me, it looks like the file is somehow corrupted. The corruption may lead from stopping or deleting the container instances/groups while testing in Azure. Since we persist this file it won’t be lost by container shutdown. Is there actually anything we can to “repair” a corrupted datafile in case we can’t avoid getting it corrupted in the first place? I can share the datafile, just tell me how I can share it with you.

I found the issue #1762 which sounds kind of similar in my opinion. Though, my datafile is not encrypted. The following stacktrace comes from the container/service running in Azure.

Stacktrace

LiteDB.LiteException: Invalid Index on 0. Full zero: True. Page Type: Empty. Prev/Next: 0/0. UniqueID: 246. ShareCounter: -1. 
   at LiteDB.Engine.IndexPage..ctor(PageBuffer buffer)
   at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
   at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at LiteDB.LiteCollection`1.Insert(T entity)

Additional context I got some more stacktraces but I think these are consequential errors. This one is also from Azure. This actually comes before the other stacktrace above in our log file but still I think that this is a consequential error too.

LiteDB.LiteException: Database lock timeout when entering in transaction mode after 00:01:00
   at LiteDB.Engine.LockService.EnterTransaction()
   at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at LiteDB.LiteCollection`1.Insert(T entity)

The following errors occured on my local development maschine when downloading the datafile from Azure and replacing it with mine. These occured when inserting an entity, but only in about 1 of 5.

System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at LiteDB.LiteCollection`1.Insert(T entity)

System.NotImplementedException: The method or operation is not implemented.
   at LiteDB.BufferSliceExtensions.ReadIndexKey(BufferSlice buffer, Int32 offset)
   at LiteDB.Engine.IndexNode..ctor(IndexPage page, Byte index, BufferSlice segment)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at LiteDB.LiteCollection`1.Insert(T entity)

System.InvalidCastException: Unable to cast object of type 'LiteDB.Engine.HeaderPage' to type 'LiteDB.Engine.IndexPage'.
   at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at LiteDB.LiteCollection`1.Insert(T entity)

System.ArgumentException: An item with the same key has already been added. Key: 512 (Parameter 'key')
   at System.Collections.Generic.SortedList`2.Add(TKey key, TValue value)
   at LiteDB.Engine.BasePage.Defrag()
   at LiteDB.Engine.BasePage.InternalInsert(UInt16 bytesLength, Byte& index)
   at LiteDB.Engine.DataPage.InsertBlock(Int32 bytesLength, Boolean extend)
   at LiteDB.Engine.DataService.<>c__DisplayClass3_0.<<Insert>g__source|0>d.MoveNext()
   at LiteDB.Engine.BufferWriter..ctor(IEnumerable`1 source)
   at LiteDB.Engine.DataService.Insert(BsonDocument doc)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at LiteDB.LiteCollection`1.Insert(T entity)

LiteDB.LiteException: Invalid Collection on 1867149421. Full zero: False. Page Type: 98. Prev/Next: 1635017028/1819045699. UniqueID: 1796. ShareCounter: -1.
   at LiteDB.Engine.CollectionPage..ctor(PageBuffer buffer)
   at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
   at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.CollectionService.Get(String name, Boolean addIfNotExists, CollectionPage& collectionPage)
   at LiteDB.Engine.Snapshot..ctor(LockMode mode, String collectionName, HeaderPage header, UInt32 transactionID, TransactionPages transPages, LockService locker, WalIndexService walIndex, DiskReader reader, Boolean addIfNotExists)
   at LiteDB.Engine.TransactionService.<CreateSnapshot>g__create|42_0(<>c__DisplayClass42_0& )
   at LiteDB.Engine.TransactionService.CreateSnapshot(LockMode mode, String collection, Boolean addIfNotExists)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.LiteCollection`1.Insert(T entity)

LiteDB.LiteException: Invalid Data on 825057890. Full zero: False. Page Type: 97. Prev/Next: 875389797/758265910. UniqueID: 1773. ShareCounter: -1.
   at LiteDB.Engine.DataPage..ctor(PageBuffer buffer)
   at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
   at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.Snapshot.GetFreeDataPage(Int32 bytesLength)
   at LiteDB.Engine.DataService.<>c__DisplayClass3_0.<<Insert>g__source|0>d.MoveNext()
   at LiteDB.Engine.BufferWriter..ctor(IEnumerable`1 source)
   at LiteDB.Engine.DataService.Insert(BsonDocument doc)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.LiteCollection`1.Insert(T entity)

I also tested this with the code currently in master and got this error:

LiteDB.LiteException: An operation that would corrupt page 3228 was prevented. The operation required 100 free bytes, but the page had only 35 available.
   at LiteDB.Engine.BasePage.InternalInsert(UInt16 bytesLength, Byte& index) in LiteDB\LiteDB\Engine\Pages\BasePage.cs:line 321
   at LiteDB.Engine.BasePage.Insert(UInt16 bytesLength, Byte& index) in LiteDB\LiteDB\Engine\Pages\BasePage.cs:line 299
   at LiteDB.Engine.DataPage.InsertBlock(Int32 bytesLength, Boolean extend) in LiteDB\LiteDB\Engine\Pages\DataPage.cs:line 47
   at LiteDB.Engine.DataService.<>c__DisplayClass3_0.<<Insert>g__source|0>d.MoveNext() in LiteDB\LiteDB\Engine\Services\DataService.cs:line 46
   at LiteDB.Engine.BufferWriter..ctor(IEnumerable`1 source) in LiteDB\LiteDB\Engine\Disk\Serializer\BufferWriter.cs:line 47
   at LiteDB.Engine.DataService.Insert(BsonDocument doc) in LiteDB\LiteDB\Engine\Services\DataService.cs:line 67
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data) in LiteDB\LiteDB\Engine\Engine\Insert.cs:line 68
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction) in LiteDB\LiteDB\Engine\Engine\Insert.cs:line 33
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn) in LiteDB\LiteDB\Engine\Engine\Transaction.cs:line 81
   at LiteDB.Engine.LiteEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId) in LiteDB\LiteDB\Engine\Engine\Insert.cs:line 20
   at LiteDB.LiteCollection`1.Insert(T entity) in LiteDB\LiteDB\Client\Database\Collections\Insert.cs:line 20

[kgday]

I don’t know if it helps but I found I was getting a similar error but and “Invalid Collection”. I had deleted the database file but left the log file that went with it. Once I deleted the log file, I was no longer having a problem. Is it possible that these errors could be caused by a corrupt or out of sync log file? Excuse my complete ignorance please if I am barking up the wrong tree!.

[Codelisk]

Is there any workaround for this?