Exception when concurrent insert (raspberry pi)

Hi

I really love LiteDb. it is easy to use.

I use LiteDb 4.1.0 with .net Core WebApi project at raspberry pi. Mostly works fine.

But when i use ab stresstest tool to test concurrent insert. ab -n 1000 -c 100 192.168.0.101/api/test/insert

it sometimes (random) throw exception (in window environment no problem) : The important thing is that once this exception throw, the insertion will become very slow and not recover even I restart the application. I need to delete the database then create again to back to normal speed.

System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Object.GetType()
   at LiteDB.BsonMapper.<>c__DisplayClass44_0.<RegisterDbRefItem>b__0(Object obj, BsonMapper m)
   at LiteDB.BsonMapper.SerializeObject(Type type, Object obj, Int32 depth)
   at LiteDB.BsonMapper.Serialize(Type type, Object obj, Int32 depth)
   at LiteDB.BsonMapper.ToDocument(Type type, Object entity)
   at LiteDB.BsonMapper.ToDocument[T](T entity)
   at LiteDB.LiteCollection`1.Insert(T document)
   at LiteDbTest.Controllers.ValuesController.Get() in C:\Users\Marcus\Documents\Visual Studio 2017\Projects\LiteDbTest\LiteDbTest\Controllers\ValuesController.cs:line 37
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()

My code in test controller:

//LiteDbManager.Instance  singleton pattern

static LiteCollection<Employee> ecol = LiteDbManager.Instance.GetCollection<Employee>("employees");
static LiteCollection<Accesslog> acol = LiteDbManager.Instance.GetCollection<Accesslog>("accesslogs");


[HttpGet]
public string Get()
{
                        //random employeeId  for test
			int value = random.Next(0, 998);
			string employeeId = $"B0{value.ToString().PadLeft(3, '0')}";

			var employee = ecol.FindById(employeeId); 

			Accesslog alog = new Accesslog()
			{
				Employee = employee ,   //BsonRef
				//Employee = new Employee() { Id = employeeId },   //<-- no exception if create a new Employee Object with Id 
				LogTime = DateTime.Now
			};

			var id = acol.Insert(alog);
			return id.AsString;

}

My Models:

public class Accesslog
{
		public Guid Id { get; set; } = new Guid();

		[Required]
		public DateTime LogTime { get; set; }

		[BsonRef("employees")] 
		public Employee Employee { get; set; }
}

public class Employee
{
		[JsonProperty("EmployeeId")]
		public string Id { get; set; }

		[StringLength(20)]
		public string LastName { get; set; }

		[StringLength(20)]
		public string FirstName { get; set; }
}

What i found is , if i create a new Employee Object only with an Id instead of using the Employee object returned by collection.FindById(employeeId) for reference insert, like below:

Accesslog alog = new Accesslog() { Employee = new Employee() { Id = employeeId },
LogTime = DateTime.Now }

then No exception will throw and work fine.

what is the correct way to Insert/Update/Upsert a document with reference.

Thanks

[mbdavid]

Next weekend (6~7/jan). Need run some unit tests and update release notes.

[mbdavid]

Hi @leung85, thanks!! The performance issues was already fixed in this commit:

https://github.com/mbdavid/LiteDB/commit/ad720af8e046c4beee46ecaeca28c5eb6c7946e3

It’s about witch Flush method use in FileStream. Using Flush(true) .net ignores OS disk flush system and do your own flush instantly.

About this bug on null in Id, it’s was a trick 😃 - I was working in v5 and got a random error that I was going crazy!! In v4.1 I changed BsonRead class to single instance per LiteEngine instance but I forgot an class variable declaration (at end of file… I never do this!)… so there are re-use this variable an, some concurrency cases, can override: https://github.com/mbdavid/LiteDB/commit/83f193ae5107b81ac58dca1e467db29455cbb494