ElasticClient.Search is not respecting SourceSerializer on results

NEST/Elasticsearch.Net version: v6.0.2 NEST/JsonNetSerializer: v6.0.2

Elasticsearch version: 6.2.2 (Docker container)

Description of the problem including expected versus actual behavior: Using a custom SourceSerializer, specified via the ConnectionSettings, to control DateTimeOffset serialization. The test object contains two DateTimeOffset fields. One field contains a value with offset -0400 (local) while the other an offset of +0000 (utc). After retrieving the records via Search, the ISearchResponse<>.Documents show that both DateTimeOffset values have been adjusted to local time.

Adding EnableDebugMode to the ConnnectionSettings and inspecting ApiCall.ResponseBodyInBytes shows that the json correctly specifies the offset values for both fields.

Further, by using ElasticClient.SourceSerializer.SerializeToString and ElasticClient.SourceSerializer.Deserialize directly, both fields are converted correctly.

Based on this, it appears that Search is not using the SourceDeserializer or not using the injected JsonSerializerSettings

Steps to reproduce:

1. Create a custom serializer that preserves the offset value in DateTimeOffset

public sealed class ElasticSearchSerializer : ConnectionSettingsAwareSerializerBase
    {
        public ElasticSearchSerializer(IElasticsearchSerializer builtinSerializer, IConnectionSettingsValues connectionSettings)
            : base(builtinSerializer, connectionSettings)
        {
        }

        protected override JsonSerializerSettings CreateJsonSerializerSettings()
        {
            return new JsonSerializerSettings()
            {
                DateFormatHandling = DateFormatHandling.IsoDateFormat,
                DateParseHandling = DateParseHandling.DateTimeOffset,
                DateTimeZoneHandling = DateTimeZoneHandling.RoundtripKind,
                Formatting = Formatting.Indented
            }
        }

        protected override void ModifyContractResolver(ConnectionSettingsAwareContractResolver resolver)
        {
            resolver.NamingStrategy = new CamelCaseNamingStrategy();
        }
    }

1. Setup ConnectionSettings & Client

            ConnectionSettings connectionSettings = new ConnectionSettings(pool, new HttpConnection(), (builtin, values) => new ElasticSearchSerializer(builtin, values));
            connectionSettings.DefaultMappingFor(typeof(TestEntity), desc => desc.IndexName("entity")).EnableDebugMode();
            ElasticClient client = new ElasticClient(connectionSettings);

1. Setup a TestEntity

        private class TestEntity
        {
            public Guid Uuid { get; set; } = new Guid("12345678-aaaa-bbbb-cccc-ddddeeeeffff");

            public string Str { get; set; } = "Hello World";

            public int Int { get; set; } = 2000;

            public double Dbl { get; set; } = 3.1415;

            public bool TrueFalse { get; set; } = true;

            public DateTime DtLocal { get; set; } = new DateTime(2018, 2, 1, 15, 0, 0, DateTimeKind.Local);

            public DateTime DtUtc { get; set; } = new DateTime(2018, 4, 1, 0, 0, 0, DateTimeKind.Utc);

            public DateTime DtUnspec { get; set; } = new DateTime(2018, 6, 1, 15, 0, 0, DateTimeKind.Unspecified);

            public DateTimeOffset DtoLocal { get; set; } = new DateTime(2018, 8, 1, 15, 0, 0, DateTimeKind.Local);

            public DateTimeOffset DtoUtc { get; set; } = new DateTime(2018, 10, 1, 0, 0, 0, DateTimeKind.Utc);
        }

1. Index and Search for the entity

            client.Index(entity, descriptor => descriptor.Id(entity.Uuid.ToString("N")).Index("entity"));
            ISearchResponse<TestEntity> result = client.Search<TestEntity>(s => s.Query(q => q.MatchAll()));

1. Inspect the ‘result’ object. Look at the result.ApiCall.ResponseBodyInBytes & result.Documents. Notice the offset value in the json and what is shown in the document. The dtoUtc will logically have the correct value but will have been converted to the local offset.

2. Test the serializer directly. Notice that the offset remains correct.

            string json = client.SourceSerializer.SerializeToString(entity);
            TestEntity deserialized;
            using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
            {
                deserialized = serializer.Deserialize<T>(stream);
            }

For anyone else encountering this issue, it can be worked around by using EnableDebugMode and manually processing response json.

private IEnumerable<T> ManualDeserialize(ISearchResponse<T> resp)
        {
            if (!resp.IsValid)
                return null;

            if (resp.ApiCall?.ResponseBodyInBytes == null)
                throw new Exception("ElasticSearch connection settings must include 'EnableDebugMode' or the json serialization workaround fails");

            // Get the raw response JSON
            string respJson = Encoding.UTF8.GetString(resp.ApiCall?.ResponseBodyInBytes);

            // Parse it generically
            JObject rootObj = (JObject)JsonConvert.DeserializeObject(respJson, {see above for JsonSerializerSettings});

            // NOTE: This is fragile; json format may change (Assumes Newtonsoft.JSON 11.0.1)
            // Check for any matches
            int hitCount = (int)rootObj.SelectToken("$.hits.total");
            TDto[] dtos = new TDto[hitCount];

            // Process each 'hit' individually due to the actual json being nested down a few levels
            for (int i = 0; i < hitCount; ++i)
            {
                // Grab json for a result and deserialized
                string hitJson = rootObj.SelectToken($"$.hits.hits[{i}]._source").ToString();
                dtos[i] = JsonConvert.DeserializeObject<T>(hitJson,  {see above for JsonSerializerSettings});
            }

            return dtos;
        }

[russcam]

I’ve opened https://github.com/elastic/elasticsearch-net/pull/3278 to address

[WereDev]

Thank you!