[BUG] Intermittent Invalid IndexPage/DataPage buffer on 0

Version Which LiteDB version/OS/.NET framework version are you using. (REQUIRED) 5.0.1. Here’s the full crash report for a load, and then a crash report for a save: Framework 4.7

LOAD (DataPage error)

Package: Acars
Version: 1.0.0.823
OS: WINDOWS
Windows: 10.0.18363
Date: 2020-02-21T02:52:10.0507405Z

Exception Stack:
LiteDB.LiteException: Invalid DataPage buffer on 0
  at LiteDB.Engine.DataPage..ctor(PageBuffer buffer)
  at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
  at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.DataService.<Read>d__5.MoveNext()
  at LiteDB.Engine.BufferReader..ctor(IEnumerable`1 source, Boolean utcDate)
  at LiteDB.Engine.DatafileLookup.Load(PageAddress rawId)
  at LiteDB.Engine.BasePipe.<LoadDocument>d__6.MoveNext()
  at LiteDB.Engine.QueryPipe.<Select>d__2.MoveNext()
  at LiteDB.Engine.QueryExecutor.<>c__DisplayClass10_0.<<ExecuteQuery>g__RunQuery|0>d.MoveNext()
  at LiteDB.BsonDataReader..ctor(IEnumerable`1 values, String collection)
  at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
  at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
  at LiteDB.LiteQueryable`1.<ToDocuments>d__26.MoveNext()
  at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
  at Acars.SimObjects.AirportDatabase.Load(String folder, SimType type)
  at Acars.SimObjects.AirportDatabase.Load()
  at Acars.MainWindow.<metroWindow_ContentRendered>d__26.MoveNext()
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.DispatcherOperation.InvokeImpl()
  at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
  at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Windows.Threading.DispatcherOperation.Invoke()
  at System.Windows.Threading.Dispatcher.ProcessQueue()
  at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
  at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
  at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
  at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
  at System.Windows.Application.RunDispatcher(Object ignore)
  at System.Windows.Application.RunInternal(Window window)
  at Acars.App.Main()

Looks like there were 9+ crashes on this one. The associated code (extra stuff removed).

lock (_lock) {
    var dbPath = GetDbName(type);

    using (var db = new LiteDatabase(dbPath)) {
        var airports = db.GetCollection<FsAirport>("airports");
        var apts = airports.FindAll();

        foreach (var apt in apts) {
            Airports.Add(apt);
        }
    }
}

Then on saving:

Package: Acars
Version: 1.0.0.821
OS: WINDOWS
Windows: 10.0.18363
Date: 2020-02-21T01:17:29.4764536Z

Exception Stack:
LiteDB.LiteException: Invalid IndexPage buffer on 0
  at LiteDB.Engine.IndexPage..ctor(PageBuffer buffer)
  at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
  at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
  at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
  at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
  at LiteDB.Engine.LiteEngine.<>c__DisplayClass30_0.<Upsert>b__0(TransactionService transaction)
  at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
  at LiteDB.LiteCollection`1.Upsert(T entity)
  at Acars.SimObjects.AirportDatabase.Save()
  at Acars.ViewModels.SettingsViewModel.<ResyncScenery>b__37_0()
  at System.Threading.Tasks.Task.Execute()
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at Acars.ViewModels.SettingsViewModel.<ResyncScenery>d__37.MoveNext()
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.DispatcherOperation.InvokeImpl()
  at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
  at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Windows.Threading.DispatcherOperation.Invoke()
  at System.Windows.Threading.Dispatcher.ProcessQueue()
  at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
  at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
  at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
  at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
  at System.Windows.Application.RunDispatcher(Object ignore)
  at System.Windows.Application.RunInternal(Window window)
  at Acars.App.Main()

This save code is:

lock (_lock) {
    var dbFile = GetDbName(SimType);

    // Delete the old db file. Don't bother with updating rows
    if(File.Exists(dbFile)) {
        File.Delete(dbFile);
    }

    using (var db = new LiteDatabase(dbFile)) {
        var airports = db.GetCollection<FsAirport>("airports");
        var runways = db.GetCollection<FsRunway>("runways");
        var gates = db.GetCollection<FsGate>("gates");

        airports.EnsureIndex("ICAO");
        runways.EnsureIndex("ICAO");
        gates.EnsureIndex("ICAO");

        //airports.InsertBulk(Airports);
        foreach(var apt in Airports) {
            airports.Upsert(apt);

            if (apt.Runways.Count > 0) {
                runways.InsertBulk(apt.Runways);
            }

            if (apt.Gates.Count > 0) {
                gates.InsertBulk(apt.Gates);
            }
        }
    }
}

I do have this run code, but it’s hard to tell if it’s running from within this - it runs a Save() and then a Load() to switch to the newer database.

await Task.Run(() => {
    _airportDatabase.CreateDatabase(_appStateManager.Settings.SimType);

    // If they changed the SimType, reload the airport database
    if (_airportDatabase.SimType != _appStateManager.Settings.SimType) {
        _airportDatabase.Load();
    }
});

So I am deleting the old database file before saving. Maybe this isn’t the best idea. But this code hasn’t changed since using 4.x

[nabeelio]

@lbnascimento are you the new maintainer? Any ideas on this issue? It’s a huge problem

[t00]

@botne Seeing these issues cropping up I am glad I decided to switch to Dapper+PostgreSQL eventually, LiteDB is not stable for a server workload, even if the size of data is small like my simple internal app running on t3.micro. I suppose it has it’s uses in the desktop apps though, with necessary precautions.