question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

NEST suddenly starting to create incorrect requests

See original GitHub issue

This issue was originally posted on https://discuss.elastic.co because I can not for sure say that it is caused by a problem in NEST. But @russcam asked me to post here as well, so here goes.

We have a .NET application using NEST version 2.3.1 against Elasticsearch 2.2.0. The application is a backend server application running on multiple hosts under IIS, all performing the same queries against an Elasticsearch cluster.

Recently we experienced very strange behavior at one of our customer sites running the application on 4 IIS hosts. Suddenly all queries against Elasticsearch from one of the nodes started to fail, the other 3 nodes continued to work fine.

The error reported from Elasticsearch for queries from the failing node was the following:

StatusCode: 400, Method: POST, Url: https://--the-url--/_search?timeout=5s, Elasticsearch reason: 
[term] query does not support [has_child], Exception: System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)

After enabling verbose logging of the request on the application side we found that we actually did send the following query in the request:

{"query":{"bool":{"filter":[{"term":{"the_property_name":{"has_child":"4"}}}]}}}

This is obviously an incorrect query, so the Elasticsearch response is correct. The query should look like this (with “value” instead of “has_child”):

{"query":{"bool":{"filter":[{"term":{"the_property_name":{"value":"4"}}}]}}}

From the C# code of the application we just perform the following for the term condition using the NEST client:

Query<T>.Term("the_property_name", "4")

It appears that the NEST client suddenly started writing incorrect requests using “has_child” instead of “value” as the property name for the condition value. After restarting the process (by performing an IIS recycle of the application pool) the problem disappeared.

We took a memory dump before performing the recycling, and I have analyzed it.

NEST has a static cache (TypeExtensions._cachedTypeProperties) which contains a mapping from types (like e.g. TermQuery) to a list of JsonProperty objects, each describing a property of the type. Among other things the JsonProperty contains PropertyName (which is the name used in the serialized json, typically set with a JsonProperty attribute) and UnderlyingName (which is the name of backing field/property in the type).

The TermQuery class implements the ITermQuery interface:

public interface ITermQuery : IFieldNameQuery
{
	[JsonProperty("value")]
	object Value { get; set; }
}

Meaning that the JsonProperty for the Value should have PropertyName=“value” and UnderlyingName=“Value”.

But listing all JsonProperty objects for the TermQuery in WinDbg gives the following:

[0] 000000040ef31110 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f1bb9f8 (System.String) Length=13, String="conditionless"
    <UnderlyingName>k__BackingField:000000050f0bffa0 (System.String) Length=13, String="Conditionless"
[1] 000000040ef31238 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f1bbaa0 (System.String) Length=5, String="_name"
    <UnderlyingName>k__BackingField:000000060eebd5c8 (System.String) Length=4, String="Name"
[2] 000000040ef31360 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f1bbb38 (System.String) Length=5, String="boost"
    <UnderlyingName>k__BackingField:000000060eebd5f0 (System.String) Length=5, String="Boost"
[3] 000000040ef31488 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f0d07b8 (System.String) Length=10, String="isVerbatim"
    <UnderlyingName>k__BackingField:000000050f0bffd8 (System.String) Length=10, String="IsVerbatim"
[4] 000000040ef315b0 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f0d0478 (System.String) Length=8, String="isStrict"
    <UnderlyingName>k__BackingField:000000060eebd618 (System.String) Length=8, String="IsStrict"
[5] 000000040ef316d8 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f0d0618 (System.String) Length=10, String="isWritable"
    <UnderlyingName>k__BackingField:000000060eebd648 (System.String) Length=10, String="IsWritable"
[6] 000000040ef31800 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000070f1bbca8 (System.String) Length=5, String="field"
    <UnderlyingName>k__BackingField:000000070f1baf70 (System.String) Length=5, String="Field"
[7] 000000040ef31928 (Newtonsoft.Json.Serialization.JsonProperty)
    _propertyName:000000050f0b7078 (System.String) Length=9, String="has_child"
    <UnderlyingName>k__BackingField:000000040ef309f0 (System.String) Length=5, String="Value"

Notice the last one. Underlying name is “Value”, but the property name is “has_child”. So this is what is causing the incorrectly generated requests.

I have browsed the source code of NEST/Elasticsearch.Net, looking at how the serialization of queries is performed. Including trying to find e.g. thread unsafety anywhere in FieldNameQueryJsonConverter, ReserializeJsonConverter and the extension method GetCachedObjectProperties. But I have not found anything that looks incorrect.

Has anyone experienced anything similar? Any idea on what could cause this other than some sort of memory corruption? We do run unmanaged code in the same process, but to me it is kind of too much of a coincidence that memory corruption should replace that “value” string with “has_child” which is also a valid Elasticsearch string set as JsonProperty attribute within the NEST client.

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
Mpdreamzcommented, Oct 30, 2017

I have fixed this as per #2883 and pushed NEST 2.5.6 to nuget.org that contains the fix:

https://www.nuget.org/packages/NEST/2.5.6

Thanks again for raising this @hmanner 👍

0reactions
hmannercommented, Oct 30, 2017

@Mpdreamz Looks like a very probable cause of the problem. Good find, thank you!

Read more comments on GitHub >

github_iconTop Results From Across the Web

NEST suddenly starting to create incorrect requests
It appears that the NEST client suddenly started writing incorrect requests using "has_child" instead of "value" as the property name for ...
Read more >
Troubleshoot Nest app sign-in issues
If you're signing in with a non-migrated Nest Account and you have trouble signing in, you may get a message saying “incorrect email...
Read more >
The most common Google Home problems, and how to fix ...
Google Home and Nest devices are smart, but they aren't perfect: Here are the most common issues, and what solutions we've found work...
Read more >
Nest Authorization Error Error 400: invalid_request
Nest Authorization Error Error 400: invalid_request ... Well, I figured it was time to move my nest account and start using the new...
Read more >
Error Messages | Nest Developers
Invalid thermostat ID. The thermostat_id is invalid. Parameters. HTTP Status Code: 400 Bad Request.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found