Issues with NEST 7. “The stream with Id and Tag is disposed."

Initially asked question here but as mentioned would be better to track it on github.

NEST/Elasticsearch.Net version: 7.3.1 .net core 2.0 on centos7

Elasticsearch version: 7.3.2 basic on centos7

Description of the problem including expected versus actual behavior:

After update to Nest7 (basically nuget, no other changes) occasionally getting this error for a process that usually runs for 15 minutes. Sometimes it happens during updating documents by id, sometimes bulk operations. The problem disappeared when I turned off parallel processing ParallelOptions { MaxDegreeOfParallelism = 1 } so, basically lowered the load. Have never had the issue with Nest6. IElasticClient is registered as singleton via DI. The client talks directly to ES by IPs set in StaticConnectionPool

System.AggregateException: One or more errors occurred. (The read operation failed, see inner exception.) ---> Elasticsearch.Net.UnexpectedElasticsearchClientException: The read operation failed, see inner exception. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ObjectDisposedException: Cannot access a disposed object.

Object name: 'The stream with Id f434e91e-8fcd-4051-83fd-56ca445e5e53 and Tag is disposed.'.
at Elasticsearch.Net.RecyclableMemoryStream.CheckDisposed()
at Elasticsearch.Net.RecyclableMemoryStream.SafeRead(Byte[] buffer, Int32 offset, Int32 count, Int32& streamPosition)
at Elasticsearch.Net.RecyclableMemoryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.MemoryStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)

---> (Inner Exception #0) Elasticsearch.Net.UnexpectedElasticsearchClientException: The read operation failed, see inner exception. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ObjectDisposedException: Cannot access a disposed object.

--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.MultiAgent.TransferDataFromRequestStream(IntPtr buffer, Int32 length, EasyRequest easy)
at System.Net.Http.CurlHandler.MultiAgent.CurlSendCallback(IntPtr buffer, UInt64 size, UInt64 nitems, IntPtr context)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.CurlResponseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Nest.ElasticClient.DoRequest[TRequest,TResponse](TRequest p, IRequestParameters parameters, Action`1 forceConfiguration)
at Nest.ElasticClient.Search[TDocument](ISearchRequest request)
at Nest.ElasticClient.Search[TDocument](Func`2 selector)
at Loader.Writers.CustomerRepository.SearchCustomer(Func`2 query)
at Loader.Loaders.EmailSentLoader.GetUpdatedCollection(DealerDto dealer, EmailSent[] batch, ConcurrentDictionary`2 bulkCollection)
at Loader.Loaders.EmailSentLoader.<>c__DisplayClass9_1.<Load>b__1(Int32 page)
at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)

without DisableDirectStreaming() stacktrace looks a bit different:

--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.MultiAgent.TransferDataFromRequestStream(IntPtr buffer, Int32 length, EasyRequest easy)
at System.Net.Http.CurlHandler.MultiAgent.CurlSendCallback(IntPtr buffer, UInt64 size, UInt64 nitems, IntPtr context)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Net.Http.CurlHandler.CurlResponseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Elasticsearch.Net.Utf8Json.JsonSerializer.FillFromStream(Stream input, Byte[]& buffer)
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Loader.Writers.CustomerRepository.SearchCustomer(Func`2 query)
...

Both exceptions led me here and here.

Seems a problem related to CurlResponseStream and libcurl.

That’s what i see on the box:

# yum list installed
...
libcurl.x86_64    7.29.0-46.el7    @base
dotnet-host.x86_64    2.1.5-1    @packages-microsoft-com-prod
dotnet-hostfxr-2.0.5.x86_64	2.0.5-1    @packages-microsoft-com-prod
dotnet-runtime-2.0.5.x86_64    2.0.5-1    @packages-microsoft-com-prod
dotnet-sdk-2.1.4.x86_64    2.1.4-1    @packages-microsoft-com-prod
...
# yum list available libcurl
libcurl.x86_64    7.29.0-54.el7    @base

As @russcam mentioned Nest6 as well as Nest7 (.net core >=2.1) uses SocketsHttpHandler. Checked if UseSocketsHttpHandler is set to false. It’s not.

Currently checking if the app converted to .net core 2.2 + updated libs on AMI will fix the problem.

Will let you know.

Steps to reproduce: short/simplified example looks like this let me know if you need better example. I’ll try to get something even closer if this is not enough. Some numbers according to logs during stress tests:

If this process runs for two hours it can fetch 3,712,029 documents from ES and around 95% of them will be sent for bulk update. Max response size (300 docs) for this loader is about 6MB.

Two other loaders that have the same structure also get exception on Search. but there is one where it happens on Update.

Provide ConnectionSettings (if relevant):

connectionSettings.EnableHttpCompression();
#if DEBUG
//connectionSettings.EnableDebugMode();
#endif
connectionSettings.DisableDirectStreaming();
connectionSettings.EnableHttpPipelining();
connectionSettings.PrettyJson(false);
connectionSettings.IncludeServerStackTraceOnError();

[russcam]

That’s good to hear @andrx. Would still liked to dig into this but it’s unlikely to be this week

[andrx]

@kuleshov-aleksei good luck