AsyncLocal issues within `IJobFactory`

Describe the bug I am having issues with AsyncLocal<T> created within an IJobFactory. I have a IJob which needs to run in a specific context (tenant) and a lot of services depend on the tenant being set. I’ve done this so far by setting an object as AsyncLocal containing the tenant information.

The object itself is set early in the process (i.e. middleware before the controllers, after receiving an RabbitMQ message and just after creating the scoped provider and on workflow engine calls just after the scope has been instantiated).

This has been working good so far. For several reasons a scoped service can’t be used here. One of the reasons being it can’t be used in singletons which are instantiated once and handle multiple requests). So there a similar approach to IHttpContextAccessor in ASP.NET Core is used by storing the object and its context specific information within it and it gets passed down with all and every async execution and has the advantage that you can change the context for a specific async execution (i.e. run code on a different tenant) without affecting the parents context.

The problem arises with Quartz.NET. After having a close look I thought IJobFactory would be the proper place to extend it, since that’s where the scope is created. So I extended MicrosoftDependencyInjectionJobFactory and override ConfigureScope and SetObjectProperties properties.

My IJobFactory looks like this

public class CustomContextJobFactory : MicrosoftDependencyInjectionJobFactory, IJobFactory
{
	private readonly ICustomContextAccessor customContextAccessor;
	private readonly ILogger<CustomContextJobFactory> logger;

	public CustomContextJobFactory(
		IServiceProvider serviceProvider,
		ICustomContextAccessor customContextAccessor,
		ILogger<CustomContextJobFactory> logger,
		IOptions<QuartzOptions> options
	): base(serviceProvider, options)
	{
		this.customContextAccessor = customContextAccessor ?? throw new ArgumentNullException(nameof(customContextAccessor));
		this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
	}

	protected override void ConfigureScope(IServiceScope scope, TriggerFiredBundle bundle, IScheduler scheduler)
	{
		base.ConfigureScope(scope, bundle, scheduler);
		ICustomContextFactory customContextFactory = scope.ServiceProvider.GetService<ICustomContextFactory>();

		ICustomContext customContext = customContextFactory.Create();
	}

	public override void SetObjectProperties(object obj, JobDataMap data)
	{
		base.SetObjectProperties(obj, data);
		if (data is not null && data.ContainsKey(CustomContext.TenantIdKey))
		{
			string tenantId = data.GetString("tenantId");
			ICustomContext context = customContextAccessor.CustomContext;
			
			context.AddHeader(CustomContext.TenantIdKey, tenantId);
			context.AddHeader(CustomContext.TenantKey, Encoding.Unicode.GetBytes(tenantId));
		}
	}
}

ConfigureScope would create the context object right after the scope has been created and SetObjectProperties would set the tenant to the newly created context. So far so good.

The problem is, when the IJobexecutes, the injected ICustomContextAccessor.CustomContext is null. This seems to be because the IJobShell is not instantiating and executing in the same async context.

Imagine this task/async continuations

Parent Task |-> IJobShell.Initialize |- IJobFactory.NewJob <- AsyncLocal created here |-> IJobShell.Run |- IJob.Execute <- AsyncLocal null here

The behavior is expected, since the AsyncLocal was created within an async context and after leaving that context the previous state will be restored.

The problem here is, that IJob.Execute is called in a different async execution which is run by the task calling IJobShell.Initialize, so the AsyncLocal created when the job is created can’t flow in when the job executes.

If the job would be executed directly after being created (or in an task executed by initialize), then the AsyncLocal would be passed properly through the async execlutions, i.e

Parent Task |-> IJobShell.InitializeAndStart |- IJobFactory.NewJob <- AsyncLocal created here |- IJob.Execute <- AsyncLocal null here

|-> async/Task executions |- synchronous executions

A clear and concise description of what the bug is.

Version used Quartz.Extensions.Hosting: 3.3.3

To Reproduce

public interface ICustomContextAccessor
{
	ICustomContext? CustomContext { get; set; }
}

public class CustomContextAccessor : ICustomContextAccessor
{
	private static readonly AsyncLocal<CustomContextHolder> _customContextCurrent = new();
	public ICustomContext? CustomContext
	{
		get => _customContextCurrent.Value?.Context;
		set
		{
			var holder = _customContextCurrent.Value;
			if(holder is not null)
			{
				holder.Context = null;
			}

			if(value is not null)
			{
				_customContextCurrent.Value = new CustomContextHolder { Context = value };
			}
		}
	}

	private class CustomContextHolder
	{
		public ICustomContext? Context;
	}
}
	
public class DefaultCustomContextFactory : ICustomContextFactory
{
	private readonly ICustomContextAccessor? customContextAccessor;

	public DefaultCustomContextFactory(IServiceProvider serviceProvider)
	{
		customContextAccessor = serviceProvider.GetService<ICustomContextAccessor>();
	}

	public ICustomContext Create()
	{
		CustomContext context = new();
		Initialize(context);

		return context;
	}

	public Task<ICustomContext> CreateWithTenant(string tenantId)
	{
		ICustomContext context = Create(new()
			{
				{ "Tenant", Encoding.Unicode.GetBytes(tenantId) }
			}
		);

		return Task.FromResult(context);
	}

	private void Initialize(CustomContext context)
	{
		if (customContextAccessor is not null)
		{
			customContextAccessor.CustomContext = context;
		}
	}
}

public interface ICustomContextFactory
{
	ICustomContext Create();
	Task<ICustomContext> CreateWithTenant(string tenantId);
}

public interface ICustomContext
{
	string? TenantId { get; }
	TenantDto? Tenant { get; }

	void AddHeader(string headerName, object headerValue);
	object? GetHeaderByKey(string headerKey);
}

public class CustomContext : ICustomContext
{
	public const string TenantIdKey = "TenantId";

	private readonly Dictionary<string, object> _headers;

	public string? TenantId => Tenant?.TenantId ?? GetTenantId();
	public TenantDto? Tenant => GetHeaderByKey(TenantDtoKey) as TenantDto;

	public CustomContext()
	{
		_headers = new Dictionary<string, object>();
	}

	public CustomContext(Dictionary<string, object> parameters)
	{
		_headers = new Dictionary<string, object>(parameters);
	}

	public object? GetHeaderByKey(string headerKey)
	{
		if (_headers.TryGetValue(headerKey, out object? headerValue))
		{
			return headerValue;
		}

		return null;
	}

	private string? GetTenantId()
	{
		string? tenantId = null;
		if(GetHeaderByKey(TenantIdKey) is string tenantString)
		{
			tenantId = tenantString;
		}

		return tenantId;
	}

	public void AddHeader(string headerName, object headerValue) => _headers[headerName] = headerValue;
}

Job registered with

    options.UseJobFactory<CustomContextJobFactory>();
    options.ScheduleJob<MyJobJob>(trigger => trigger
	.WithIdentity($"MyJob for Tenant 001", "AsyncLocalTest")
	.UsingJobData(CustomContext.TenantIdKey, "001")
	.WithDailyTimeIntervalSchedule(x => x.WithInterval(10, IntervalUnit.Second))
	.WithDescription($"Polls tenant '001' every 10 seconds .")
    );

Expected behavior

Expected behavior would be that AsyncLocal created in the IJobFactory are passed through to the job execution

[TsengSR]

@lahma Any idea why the AsyncLocal variable is losing its value. I am having the same scenario, passing the tenantid in the AsyncLocal variable and the value is lost after the NewJob function is executed. Is there any plan to fix this issue?

That’s because AsyncLocal only retains it’s value during that async execution, i.e. if you have a method called Task async Execute() and set AsyncLocal within it, it will be restored after you leave the Execute() method.

That’s why it needs happen very early in the flow, before the first containers are instantiated. In my other code I use it, I place it at the earlierst point possible (Tenant Middleware for Http Requests, in MessageBus implementation, right after the scoped provider was created etc.)

@lahma No, thats exactly the point of using AsyncLocal because its not tied to a thread, but to an async execution (AsyncLocal is the ThreadLocal counter part of async/await/Task methods). My only issue was that MicrosoftDependencyInjectionJobFactory keeps the creation of the scope internal, so I can’t create the factory with AsyncLocal before the job is resolved and executed. Its resolved and abstracted in the ScopedJob class which is the only one which as an async method.

Basically you’d need a async Task RunJob() method and in the first line of it, create the scope, then create the context (with the AsyncLocal) and initialize it, then resolve your services/Job, then execute the job. Then the job would have the proper tenant from the AsycnLocal and by the time RunJob finishes, the AsyncLocal would be restored to its initial value (typically null since it wasn’t initialized before). This is independent on which thread the code actually runs, something along the lines of (pseudo code):

public async Task RunJob(TriggerFiredBundle bundle)
{
    var tenantId = "Tenant-001"; // obtain it from bundle/job metadata
    var scopeFactory = provider.GetRequriedService<IServiceScopeFactory>())
    using var scope = scopeFactory.CreateScope();
    var customContextFactory = scope.GetRequriedService<ICustomContextFactory>();
    // this must be done before any service is resolved. It sets the context within the AsyncLocal
    var customContext = customContextFactory.CreateWithTenant(tenantId);
    // after this point, every service, job and code executed by this async flow (even if its static or singleton) has access to the AsyncLocal by resolving ICustomContextAccessor and gaining access to the .CustomContext property

    var job = (IJob)scope.ResolveRequired(bundle.JobDetail.JobType);
    await job.Execute(/*Job execution context here);

}
// after finishing the method, the `ICustomContextAccexxor.CustomContext` property becomes null, because the async context in which it was created/assigned has been left. 

[kolpav]

@TsengSR I just skimmed your problem so it might not be applicable to you but I am also storing tenant id inside AsyncLocal as there are some things you can’t do with recommended DI approach like manually constructing query expression inside pooled db context for global filters based on current tenant, accessing tenant id inside domain layer with 0 project references and so on. I am still not sure if I don’t have bug somewhere and there is space for refactoring but its working ok so far.

I am able to change tenant id like so using var _ = _currentTenantService.Change(tenantId); code dump:

public class CurrentTenantService : ICurrentTenantService
{
  private readonly IAsyncLocalCurrentTenantAccessor _currentTenantAccessor;

  public CurrentTenantService(IAsyncLocalCurrentTenantAccessor currentTenantAccessor)
    => _currentTenantAccessor = currentTenantAccessor;

  public Guid? TenantId
  {
    get => _currentTenantAccessor.Current;
    set => _currentTenantAccessor.Current = value;
  }

  public IDisposable Change(Guid? id)
  {
    TenantId = id;
    return new DisposeAction(() => TenantId = null);
  }

  public IDisposable ChangeAndRevert(Guid? id)
  {
    var previous = TenantId;
    TenantId = id;
    return new DisposeAction(() => TenantId = previous);
  }
}

public class DisposeAction : IDisposable
{
  private readonly Action _action;
  public DisposeAction(Action action) => _action = action;
  public void Dispose() => _action();
}

public interface IAsyncLocalCurrentTenantAccessor
{
  public Guid? Current { get; set; }
}

public class AsyncLocalCurrentTenantAccessor : IAsyncLocalCurrentTenantAccessor
{
  private readonly AsyncLocal<Guid?> _currentScope;

  public static AsyncLocalCurrentTenantAccessor Instance { get; } = new();

  private AsyncLocalCurrentTenantAccessor() => _currentScope = new();

  public Guid? Current
  {
    get => _currentScope.Value;
    set => _currentScope.Value = value;
  }
}