Concurrency issue while Querying

In LiteDB 5 it appears you cannot use the result of a query as an Enumerable to pass to Parallel.Foreach if you are performing additional query operations on the database inside of the Parallel.Foreach loop.

The error message received is “System.InvalidOperationException: ‘Collection was modified; enumeration operation may not execute.’” this is thrown on var exists = col.Exists(x => x.IdentityHash == IdentityHash && x.RunId == RunId); inside WriteObjectExists

I’m not performing any writes while this is being run. The below code can reproduce the issue with a few thousand objects in the database. I’ve tried generating the IEnumerable different ways (as shown below)

public static IEnumerable<WriteObject> GetMissingFromFirst2(string firstRunId, string secondRunId)
{
    var col = db.GetCollection<WriteObject>("WriteObjects");

    var list = new ConcurrentBag<WriteObject>();

    var wos = col.Find(x => x.RunId == secondRunId);

    Parallel.ForEach(wos, wo =>
    {
        if (!WriteObjectExists(firstRunId, wo.IdentityHash))
        {
            list.Add(wo);
        }
    });

    return list;
}

public static IEnumerable<WriteObject> GetMissingFromFirst(string firstRunId, string secondRunId)
{
    var bag= new ConcurrentBag<WriteObject>();

    var identityHashes = db.Execute($"SELECT IdentityHash FROM WriteObjects WHERE RunId = @0",
            new BsonDocument
            {
                ["0"] = secondRunId
            });

    Parallel.ForEach(identityHashes.ToEnumerable(), IdentityHash =>
    {
        if (!WriteObjectExists(firstRunId, IdentityHash["IdentityHash"].AsString))
        {
            list.Add(GetWriteObject(secondRunId, IdentityHash.AsString));
        }
    });
    return bag;
}

private static bool WriteObjectExists(string RunId, string IdentityHash)
{
    var col = db.GetCollection<WriteObject>("WriteObjects");
    var exists = col.Exists(x => x.IdentityHash == IdentityHash && x.RunId == RunId);
    return exists;
}

private static WriteObject GetWriteObject(string runId, string IdentityHash)
{
    var col = db.GetCollection<WriteObject>("WriteObjects");
    return col.FindOne(x => x.IdentityHash == IdentityHash && x.RunId == runId);
}

[mbdavid]

@gfs, nested queries will be on our roadmap!

[gfs]

@gfs, can you paste your stacktrace to check?

---> System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
at LiteDB.CacheService.GetPage(UInt32 pageID)
at LiteDB.PageService.GetPage[T](UInt32 pageID)
at LiteDB.IndexService.Find(CollectionIndex index, BsonValue value, Boolean sibling, Int32 order)
at LiteDB.QueryEquals.ExecuteIndex(IndexService indexer, CollectionIndex index)+MoveNext()
at LiteDB.LinqExtensions.<>c__DisplayClass2_0`2.<<DistinctBy>g___|0>d.MoveNext()
at LiteDB.QueryCursor.Fetch(TransactionService trans, DataService data, BsonReader bsonReader)
at LiteDB.LiteEngine.Find(String collection, Query query, Int32 skip, Int32 limit)+MoveNext()
at LiteDB.LiteEngine.Find(String collection, Query query, String[] includes, Int32 skip, Int32 limit)+MoveNext()
at LiteDB.LiteCollection`1.Find(Query query, Int32 skip, Int32 limit)+MoveNext()
at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at LiteDB.LiteCollection`1.FindOne(Query query)
at AttackSurfaceAnalyzer.Utils.DatabaseManager.<>c__DisplayClass39_0.<GetModified>b__0(WriteObject WO) in C:\Users\Gstoc\Documents\GitHub\AttackSurfaceAnalyzer\Lib\Utils\DatabaseManager.cs:line 417
at System.Threading.Tasks.Parallel.<>c__DisplayClass33_0`2.<ForEachWorker>b__0(Int32 i)
at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)`