Bug: JsonSerializationException: Self referencing loop detected with type 'Nest.Field'. Path '_source.includes'.

Calling SearchAsync in a concurrent manner while specifying a source filter on the search descriptor causes a JSON serialization exception on the Nest.Field type. This only happens when running concurrent requests, and only when doing >= 8 concurrent tasks in my .NET Core project, or when doing >= 2 concurrent tasks in my LINQPad reproduction script (so .NET Framework). It also only happens when using a source filter.

The message:

JsonSerializationException: Self referencing loop detected with type ‘Nest.Field’. Path ‘_source.includes’.

However, sometimes the exception type is UnexpectedElasticsearchClientException instead of JsonSerializationException.

The stacktrace:

at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.FieldsJsonConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.UnionJsonConverter`2.<>c__DisplayClass0_0.<WriteJson>b__1(TSecond second)   at Nest.UnionJsonConverter.WriteJson(JsonWriter writer, Object v, JsonSerializer serializer)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Elastic.Internal.JsonNet.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.InternalSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting) –

A complete LINQPad reproduction script:

async Task Main()
{
	var c = new ElasticClient();
	
	while (true)
	{
		var tasks = new List<Task>();
	
		for (var i = 0; i < 10; i++)
		{
			tasks.Add(Task.Run(async () =>
			{
				var response = await c.SearchAsync<MyObject>(s =>
				{
					s
					.Index("my_index")
					.Source(src => src.Includes(f => f.Field("any_field")));
	
					Console.WriteLine(Serialize(s, c));
	
					return s;
				});
			}));
		}
	
		await Task.WhenAll(tasks);
	}
}

public class MyObject
{
	
}

public static string Serialize(ISearchRequest request, IElasticClient client)
{
	string serialized;
	using (var stream = new MemoryStream())
	using (var streamReader = new StreamReader(stream))
	{
		client.RequestResponseSerializer.Serialize(request, stream);
		stream.Position = 0;
		serialized = streamReader.ReadToEnd();
		return serialized;
	}
}

If this somehow doesn’t fail, try increasing the concurrency from 10 to something higher. You only need NEST 6.5.1 and a working Elasticsearch on the default port with a my_index index.

This is using the NEST 6.5.1 NuGet package (also reproduces for 6.5.0).

This behavior is a real head scratcher to me. I’m correct when saying that ElasticClient is meant to be thread-safe, right? That’s how we’ve always used it.

Possibly a regression of #706?

[russcam]

I’ve opened #3618 to address.

We’re looking to put out 6.6.0 very shortly, which will contain this fix

[russcam]

Closing this as it’s now in NEST 6.6.0