Fully occupied /tmp for ASP Dot Net Core hosted on AWS Lambda is breaking API's

Description

A few of the API’s intermittently are failing due to “No Space left on device error” when trying to serialize the response for the API. The framework uses NewtonSoft JSON formatter. When this formatter is trying to access the tmp storage i.e. ephemeral storage for AWS Lambda (max memory available: 512 MB), it is failing with the below error:

> Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware - message=An unhandled exception has occurred while executing the request. error=System.IO.IOException: No space left on device : '/tmp/ASPNETCORE_e0cf2c22-e853-417a-874b-22527eb4a609.tmp'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer)
   at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNetCore.WebUtilities.PagedByteBuffer.MoveTo(Stream stream)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingWriteStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write(String value)
   at Newtonsoft.Json.Utilities.JavaScriptUtils.WriteEscapedJavaScriptString(TextWriter writer, String s, Char delimiter, Boolean appendDelimiters, Boolean[] charEscapeFlags, StringEscapeHandling stringEscapeHandling, IArrayPool`1 bufferPool, Char[]& writeBuffer)
   at Newtonsoft.Json.JsonTextWriter.WriteValue(String value)
   at Newtonsoft.Json.JsonWriter.WriteValue(JsonWriter writer, PrimitiveTypeCode typeCode, Object value)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at ZS.OE.Commons.Restful.Core.Middleware.ValidateAntiForgeryTokenMiddleware.Invoke(HttpContext context)
   at ZS.OE.Commons.Restful.Core.Middleware.PostAuthenticationMiddleware.InvokeAsync(HttpContext httpContext)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at ZS.OE.Commons.Restful.Core.StartupBase.<>c.<<Configure>b__11_1>d.MoveNext()
--- End of stack trace from previous location ---
   at ZS.OE.Commons.Restful.Core.StartupBase.<>c.<<Configure>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

I wrote an dummy API to fetch the size of the files and directories in the tmp folder and when hit (luckily the same container for which the API’s failed), two distinct responses were:

/tmp/core…NET Sockets.8\ : ~620MB

/tmp/core…NET Timer : ~600MB

Dotnet version: 6.0 AWS Lambda Max memory available: 1280MB AWS Lambda Ephemeral storage: 512MB

My codebase does not use temporary storage explicitly anywhere!

Does the .Net Framework use temporary storage (such huge size) for any purpose? Is there any specific reason why the RAM is not used instead?

Reproduction Steps

It is an intermittent issue experiencing on AWS Lambda.

Expected behavior

Newton Soft Json formatter must be able to create tmp files while formatting/serializing the response.

Actual behavior

Newton Soft Json formatter is unable to create tmp files while formatting/serializing the response as the tmp is already full.

Regression?

No response

Known Workarounds

Re-deploying our code onto the lambda i.e creating new lambda version is killing already active containers and new containers that become active come with empty /tmp storage.

Configuration

.Net Version: 6.0 Lambda internally uses Linux agents, runs on x86_64 architecture.

Other information

No response

[normj]

We have a support ticket for this issue which I’m guessing @ChSankalp is involved with. Our assumption based on the stacktrace was the Newtonsoft formatter was misbehaving some how and not cleaning up after itself in some circumstances. From a concurrency point of there is only one request being run at a time within a Lambda process. Lambda is all about horizontal scaling and spins up many different compute containers to handle the incoming requests.

The possibility of a core dump being triggered is really interesting. Looking at the configuration environments variables that @noahfalk pointed to we are not setting any of those and I doubt @ChSankalp is either. Is there anything else that would trigger a core dump?

[stephentoub]

As Noah said, this looks like a core dump, and for whatever reason it picked up the name of a thread in the process. “.NET Sockets” is the name supplied to the thread that waits on an epoll: https://github.com/dotnet/runtime/blob/1ed120651c8a9e49cc4bb95cb3e6936e3f768e0e/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs#L165