Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

'Object reference not set to an instance of an object' error after restarting

See original GitHub issue

Ever since adding hangfire, we now get the following error after restarting our k8s pods. It will eventually start after throwing this error a few times. It seems that each node will throw this error until one starts, then another will not throw and start etc… Like a node is locking something during start, and all others throw this error?

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Elsa.Activities.Temporal.Hangfire.Services.HangfireWorkflowDefinitionScheduler.<>c.<QueryRecurringJobs>b__14_0(RecurringJobDto x)
   at System.Linq.Utilities.<>c__DisplayClass1_0`1.<CombinePredicates>b__0(TSource x)
   at System.Linq.Enumerable.WhereListIterator`1.ToList()
   at Elsa.Activities.Temporal.Hangfire.Services.HangfireWorkflowDefinitionScheduler.DeleteRecurringJobs(String workflowDefinitionId)
   at Elsa.Activities.Temporal.Hangfire.Services.HangfireWorkflowDefinitionScheduler.UnscheduleAsync(String workflowDefinitionId, CancellationToken cancellationToken)
   at Elsa.Activities.Temporal.Common.Handlers.ScheduleTimers.Handle(TriggerIndexingFinished notification, CancellationToken cancellationToken)
   at MediatR.NotificationPublishers.ForeachAwaitPublisher.Publish(IEnumerable`1 handlerExecutors, INotification notification, CancellationToken cancellationToken)
   at Elsa.Services.Triggers.TriggerIndexer.IndexTriggersAsync(IWorkflowBlueprint workflowBlueprint, CancellationToken cancellationToken)
   at Elsa.Services.Triggers.TriggerIndexer.IndexTriggersAsync(IEnumerable`1 workflowBlueprints, CancellationToken cancellationToken)
   at Elsa.Services.Triggers.TriggerIndexer.IndexTriggersAsync(CancellationToken cancellationToken)
   at Elsa.StartupTasks.IndexTriggers.ExecuteAsync(CancellationToken cancellationToken)
   at Elsa.Runtime.StartupRunner.StartupAsync(CancellationToken cancellationToken)
   at Elsa.Runtime.StartupRunner.StartupAsync(CancellationToken cancellationToken)
   at Elsa.HostedServices.StartupRunnerHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.<Main>$(String[] args) in /src/Dashboard/Program.cs:line 213

Issue Analytics

State:
Created 2 months ago
Comments:11 (11 by maintainers)

Top GitHub Comments

1reaction

sfmskywalkercommented, Aug 11, 2023

Looks like I got a little break (and therefore you too I guess! 😉 ) leaving me with a small window to add some logging.

0reactions

johnwccommented, Aug 12, 2023

This is as much as I can see from the pod log.

 Elsa.Retention.HostedServices.CleanupHostedService[0] Failed to perform cleanup this time around. Next cleanup attempt will happen in 00:10:00 System.NullReferenceException: Object reference not set to an instance of an object.
    at Elsa.Activities.Temporal.Hangfire.Services.HangfireWorkflowInstanceScheduler.<>c.<QueryRecurringJobs>b__14_0(RecurringJobDto x)
    at System.Linq.Utilities.<>c__DisplayClass1_0`1.<CombinePredicates>b__0(TSource x)
    at System.Linq.Enumerable.WhereListIterator`1.ToList()
    at Elsa.Activities.Temporal.Hangfire.Services.HangfireWorkflowInstanceScheduler.DeleteRecurringJobs(String workflowInstanceId)
    at Elsa.Activities.Temporal.Hangfire.Services.HangfireWorkflowInstanceScheduler.UnscheduleAsync(String workflowInstanceId, CancellationToken cancellationToken)
    at Elsa.Activities.Temporal.Common.Handlers.UnscheduleTimers.Handle(BookmarksDeleted notification, CancellationToken cancellationToken)
    at MediatR.NotificationPublishers.ForeachAwaitPublisher.Publish(IEnumerable`1 handlerExecutors, INotification notification, CancellationToken cancellationToken)
    at Elsa.Services.Bookmarks.BookmarkIndexer.DeleteBookmarksAsync(String workflowInstanceId, CancellationToken cancellationToken)
    at Elsa.Handlers.UpdateBookmarksAndRunPostTasks.Handle(ManyWorkflowInstancesDeleted notification, CancellationToken cancellationToken)
    at MediatR.NotificationPublishers.ForeachAwaitPublisher.Publish(IEnumerable`1 handlerExecutors, INotification notification, CancellationToken cancellationToken)
    at Elsa.Persistence.Decorators.EventPublishingWorkflowInstanceStore.DeleteManyAsync(ISpecification`1 specification, CancellationToken cancellationToken)
    at Elsa.Retention.Jobs.CleanupJob.DeleteManyAsync(ICollection`1 workflowInstanceIds, CancellationToken cancellationToken)
    at Elsa.Retention.Jobs.CleanupJob.ExecuteAsync(CancellationToken cancellationToken)
    at Elsa.Retention.HostedServices.CleanupHostedService.ExecuteAsync(CancellationT