expected:'"', actual:' ', at offset:61: UnexpectedElasticsearchClientException

NEST/Elasticsearch.Net version: 7.6.1

Elasticsearch version: 7.1

Description of the problem including expected versus actual behavior: Getting UnexpectedElasticsearchClientException on sequantial requests match_all rests.

expected:‘"’, actual:’ ', at offset:61: UnexpectedElasticsearchClientException at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) at Nest.ElasticClient.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct) at Nest.ElasticClient.SearchAsync[TDocument](Func2 selector, CancellationToken ct) at GlobalSearch.Infrastructure.Helpers.DefaultNestSearchExecutor.Search[T](ValueTuple`8 searchParams) in C:_agents\r\r001_work\7\s\src\globalsearch\Infrastructure\Helpers\DefaultNestSearchExecutor.cs:line 49 at GlobalSearch.Services.SearchProcessor.SearchObjectDetails(SearchRequestDto request) in C:_agents\r\r001_work\7\s\src\globalsearch\Services\SearchProcessor.cs:line 46 at GlobalSearch.Handlers.SearchProxyHandler(SearchRequestDto request, ILambdaContext context) in C:_agents\r\r001_work\7\s\src\globalsearch\Handlers.cs:line 26 at lambda_method(Closure , Stream , Stream , LambdaContextInternal )

at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentRaw() at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver ) at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver) at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken) at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable1 statusCode, IEnumerable1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken) at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken) at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken) at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)

on match_all request after a sequential set of same requests.

Steps to reproduce:

1. Start to execute the same heavy requests.
2. After some time app starts to fire UnexpectedElasticsearchClientException.

Expected behavior A clear and concise description of what you expected to happen. Expected not to receive UnexpectedElasticsearchClientException.

Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):

[iam-black]

All this is running on AWS Elasticsearch service, maybe It was worth mentioning from the beginning.

You are probably getting Elasticsearch engine exception in response body. You should check AWS ES “Error logs” to see what’s going on.

I have encountered the same error with NEST and AWS Elasticsearch in application logs. expected:'"', actual:' ', at offset:61: UnexpectedElasticsearchClientException

Error logs in my case org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:314) [elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.4.2.jar:7.4.2] …
Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.

[journey191]

@stevejgordon @russcam Exception:

Elasticsearch.Net.UnexpectedElasticsearchClientException: expected:',', actual:'"application/pdf; version=1.7"', at offset:369

The Response JSON:

{
"took": 3,
"timed_out": false,
"_shards": {
"total": 6,
"successful": 6,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "ontall_documents",
"_id": "080cf75e-a269-ab98-9d1b-3a027eae3507",
"_score": 1,
"_source": {
"highlight": [ ],
"attachment": {
"date": "2022-03-08T08:00:18Z",
"content_type": "application/pdf",
"author": "u1",
"format": "application/pdf; version=1.7",
"modified": "2022-03-08T08:00:18Z",
"language": "lt",
"creator_tool": "WPS 文字",
"content": "参考已实施项目工作机制： 1. 双方沟通机制 （1）建立微信或 QQ 群，双方所有参与人员加入。日常沟通、文件资料等在群里共享。 双方主要人员不定期以线上会议方式交流问题和解决方案，并在会后编写会议纪要。会议纪 要上传到群文件。",
"content_length": 129
},
"id": "080cf75e-a269-ab98-9d1b-3a027eae3507",
"title": "测试文档",
"content": "JVBERi0x..."
}
}
]
}
}