Triggers not firing intermittently

Version 3.2.1 Running in net7.0 aspnet application

Scenario

Some code omitted for brevity.

    public class SubmitFleetBookingForReviewHandler
        : IRequestHandler<SubmitFleetBookingForReviewInput, SubmitFleetBookingForReviewOutput>
    {
        private readonly ILogger<SubmitFleetBookingForReviewHandler> _logger;
        private readonly IRequestAuthVariables _requestAuthVars;
        private readonly ISupplierAuthorizer _supplierAuthorizer;
        private readonly IClock _clock;
        private readonly AutoGuruDbContext _db;
        private readonly IBookingStatusTransitionValidator _bookingStatusTransitioner;
        private readonly IRequestHandler<AcceptFleetSignOnAgreementRequest, Unit> _acceptFleetSignOnAgreementHandler;
        private readonly IRequestHandler<AssessBookingBusinessRulesRequest, BookingBusinessRuleBookingAssessment> _assessBookingBusinessRulesHandler;

        public SubmitFleetBookingForReviewHandler(
            AutoGuruDbContext db,
        )
        {
            _db = db;
        }

        public async Task<SubmitFleetBookingForReviewOutput> Handle(
            SubmitFleetBookingForReviewInput request,
            CancellationToken cancellationToken)
        {
            <code here that changes the booking>

            try
            {
                _logger.LogDebug("Before 1st save. Booking Id = {bookingId} Booking Status Id = {bookingStatusId} DbContextId = {dbContextId} Lease = {lease}",
                    bookingId, booking.BookingStatusID, _db.ContextId.InstanceId, _db.ContextId.Lease);
                _logger.LogDebug("Changes before 1st save. {changes}", _db.ChangeTracker.DebugView.LongView);

                await _db.SaveChangesAsync(CancellationToken.None);

                _logger.LogDebug("After 1st save. Booking Id = {bookingId} Booking Status Id = {bookingStatusId} DbContextId = {dbContextId} Lease = {lease}",
                    bookingId, booking.BookingStatusID, _db.ContextId.InstanceId, _db.ContextId.Lease);
                _logger.LogDebug("Changes after 1st save. {changes}", _db.ChangeTracker.DebugView.LongView);

                _logger.LogInformation("Fleet booking ({bookingId}) submitted for review,", bookingId);
            }
            catch (DbUpdateConcurrencyException)
            {
                return new SubmitFleetBookingForReviewOutput(bookingId, new ConcurrentUpdateError());
            }

            <code here that changes the booking again>

            _logger.LogDebug("Before 2nd save. Booking Id = {bookingId} Booking Status Id = {bookingStatusId} DbContextId = {dbContextId} Lease = {lease}",
                bookingId, booking.BookingStatusID, _db.ContextId.InstanceId, _db.ContextId.Lease);
            _logger.LogDebug("Changes before 2nd save. {changes}", _db.ChangeTracker.DebugView.LongView);

            await _db.SaveChangesAsync(CancellationToken.None);

            _logger.LogDebug("After 2nd save. Booking Id = {bookingId} Booking Status Id = {bookingStatusId} DbContextId = {dbContextId} Lease = {lease}",
                bookingId, booking.BookingStatusID, _db.ContextId.InstanceId, _db.ContextId.Lease);
            _logger.LogDebug("Changes after 2nd save. {changes}", _db.ChangeTracker.DebugView.LongView);
        }
    }

We have a number of triggers attached to the Booking, some IBeforeSaveTrigger and some IAfterSaveTrigger.

Intermittently the triggers on Booking don’t fire during the 2nd SaveChangesAsync() in the code above.

Logs

FailedLogs.txt

SuccessLogs.txt

Differences When successful, 33 triggers are discovered. When it fails, the first 22 triggers match, but 11 are missing.

Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetSupplierPaidAt" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetAutoGuruPaidAt" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetRatesAndDiscountsForFleetBookings" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetPayerAndPaymentTerms" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Booking_BeforeSave_SetMerchantServiceFeeForFleetBooking. Booking Id = 6005225 % = 0.0350 $ = 14.37
Booking Id = 6005225 Fleet Company Id = 1 Supplier Id = 830 Booking Status Id = 33 DbContextId = fba0254d-8bb0-4555-adff-a648aa665397 Lease 26
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetMerchantServiceFeeForFleetBooking" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetLastReviewedPropertiesOfBookingTicketItems" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_SetDataFee" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.Booking_BeforeSave_ResetClaimOnAction" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.Shared.Bookings.Booking_ConsumeSupplierTimeSlot" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Booking]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.CreationTimestampingTrigger" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Abstractions.ICreationTimestamped]"
Invoking trigger: "AutoGuru.DataAccess.Triggers.UpdateTimestampingTrigger" as "EntityFrameworkCore.Triggered.IBeforeSaveTrigger`1[AutoGuru.Entities.Abstractions.IUpdateTimestamped]"

The logs show the DbContextId of the DbContext injected into the code above and the DbContextId used in the trigger discovery process (TriggerContextTracker). The same context is used (as expected) but the changes don’t seem to be detected.

We’re using the default cascade behavior and limit.

In terms of testing, it is sometimes very intermittent in that I could use JMeter to submit 1000 bookings and all are successful. In other tests, multiple could fail within the first 100 requests.

I will be doing further investigations and will update as I make discoveries.

[anthony-keller]

Thanks. Let me know if I can help with anything.

[koenbeuk]

Ok I think we isolated the issue:

The SaveChangesFailed and SaveChangesFailedAsync interceptor methods are not called when a trigger raises an exception explicitly, e.g.

public class ThrowExceptionTrigger : IBeforeSaveTrigger<Entity>
{
    public void BeforeSave(ITriggerContext<Entity> context)
        => throw new InvalidOperationException("Throwing up...");
}

Will ignore SaveChangesFailed and SavedChanges and therefore not dispose of the TriggerSession. A pooled DbContext would then hang on to the trigger session which is both a memory leak and can cause the behavior that you’re describing.

We would certainly want to make TriggerSessionSaveChangesInterceptor a ResettableService to ensure that TriggerSession will always get deposed of. In addition, We’ll need to figure out if we can make AfterSaveFailed triggers work properly again or otherwise document accordingly.

I’ll spend some time on it this weekend.