Internal error in the .NET Runtime at IP 00007FF8E1FDD149 (00007FF8E1E20000) with ASP.NET Core 2.2

Describe the bug

Getting below error msgs from Eventlog. After 5 times repeated messages, the specific application pool completely shuts down (no restarting).

Entry 1 (in event log)

Application: w3wp.exe CoreCLR Version: 4.6.27110.4 Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FF8E1FDD149 (00007FF8E1E20000) with exit code c0000005.

Entry 2

Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a Faulting module name: coreclr.dll, version: 4.6.27110.4, time stamp: 0x5be75633 Exception code: 0xc0000005 Fault offset: 0x00000000001bd149 Faulting process id: 0x172c Faulting application start time: 0x01d49964aeaaca09 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\coreclr.dll Report Id: 9693a536-e15e-4c29-bb23-eee06a89815d Faulting package full name: Faulting package-relative application ID:

Entry 3 - note: WER and TEMP folders does not contain the stated files:

Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0

Problem signature: P1: w3wp.exe P2: 10.0.14393.0 P3: 57899b8a P4: coreclr.dll P5: 4.6.27110.4 P6: 5be75633 P7: c0000005 P8: 00000000001bd149 P9: P10:

Attached files: \?\C:\Windows\Temp\WER7FC4.tmp.appcompat.txt \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8022.tmp.WERInternalMetadata.xml WERGenerationLog.txt

These files may be available here: C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_9d888ac04d24e4ae9ba23c5f7762b4ddf95248f_ed19be39_cab_0b048031

Analysis symbol: Rechecking for solution: 0 Report Id: 9693a536-e15e-4c29-bb23-eee06a89815d Report Status: 131076 Hashed bucket:

Entry 4

Fault bucket 2071344263271118212, type 4 Event Name: APPCRASH Response: Not available Cab Id: 0

Problem signature: P1: w3wp.exe P2: 10.0.14393.0 P3: 57899b8a P4: coreclr.dll P5: 4.6.27110.4 P6: 5be75633 P7: c0000005 P8: 00000000001bd149 P9: P10:

Attached files: \?\C:\Windows\Temp\WER7FC4.tmp.appcompat.txt \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8022.tmp.WERInternalMetadata.xml WERGenerationLog.txt

These files may be available here: C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_w3wp.exe_9d888ac04d24e4ae9ba23c5f7762b4ddf95248f_ed19be39_11f88503

Analysis symbol: Rechecking for solution: 0 Report Id: 9693a536-e15e-4c29-bb23-eee06a89815d Report Status: 0 Hashed bucket: ba850d50d8e1d2fd9cbee4a20bd17584

Now after 4th entry log, I see the specific application pool is restarted. This entire process happens 3 times in total - in about 2-3 minutes time total. In this particular case, 8 hours pass and we see a similar process repeating again.

To Reproduce

Steps to reproduce the behavior: Using ASP.NET Core 2.2 on Windows 2016 Server. Running InProcess Site runs normally but suddenly we get this error - don’t know what code triggers the error. From web.config:

<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" requireAccess="Script" />
    </handlers>
    <aspNetCore processPath=".\myproject.exe" arguments="exec &quot;C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll&quot;" stdoutLogEnabled="true" stdoutLogFile=".\logs\log" hostingModel="InProcess">
      <handlerSettings>
        <handlerSetting name="debugLevel" value="file" />
        <handlerSetting name="debugFile" value=".\logs\ancm.log" />
      </handlerSettings>
      <environmentVariables>
        <environmentVariable name="ASPNETCORE_HTTPS_PORT" value="443" />
        <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Production" />
      </environmentVariables>
    </aspNetCore>

We have a bunch of these errors in the stdout log - but they have been there in ASP.NET Core 2.1.x as well:

warn: Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery[8]
      The 'Cache-Control' and 'Pragma' headers have been overridden and set to 'no-cache, no-store' and 'no-cache' respectively to prevent caching of this response. Any response that uses antiforgery should not be cached.

FROM ANCM log

[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\sites\www.myproject.com' [aspnetcorev2_inprocess.dll] Starting file watcher thread [aspnetcorev2.dll] Initializing logs for ‘C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll’. Process Id: 2480… File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2. Commit: ce8cf65589734f82b0536c543aba5bd60d0a5a98. [aspnetcorev2.dll] Resolving hostfxr parameters for application: ‘.\myproject.exe’ arguments: ‘exec “C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll”’ path: 'C:\inetpub\sites\www.myproject.com' [aspnetcorev2.dll] Known dotnet.exe location: ‘’ [aspnetcorev2.dll] Process path ‘.\myproject.exe’ is not dotnet, treating application as standalone or portable with bootstrapper [aspnetcorev2.dll] Checking application.dll at ‘C:\inetpub\sites\www.myproject.com.\myproject.dll’ [aspnetcorev2.dll] Checking hostfxr.dll at ‘C:\inetpub\sites\www.myproject.com.\hostfxr.dll’ [aspnetcorev2.dll] hostfxr.dll found app local at ‘C:\inetpub\sites\www.myproject.com.\hostfxr.dll’, treating application as standalone [aspnetcorev2.dll] Parsed hostfxr options: dotnet location: ‘’ hostfxr path: ‘C:\inetpub\sites\www.myproject.com.\hostfxr.dll’ arguments: [aspnetcorev2.dll] Argument[0] = ‘C:\inetpub\sites\www.myproject.com.\myproject.exe’ [aspnetcorev2.dll] Argument[1] = ‘exec’ [aspnetcorev2.dll] Argument[2] = ‘C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll’ [aspnetcorev2.dll] c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020 [aspnetcorev2.dll] Loading request handler: ‘C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.0\aspnetcorev2_inprocess.dll’ [aspnetcorev2.dll] Creating handler application [aspnetcorev2_inprocess.dll] Initializing logs for ‘C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.0\aspnetcorev2_inprocess.dll’. Process Id: 2480… File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: ce8cf65589734f82b0536c543aba5bd60d0a5a98. [aspnetcorev2_inprocess.dll] Waiting for initialization [aspnetcorev2_inprocess.dll] Starting in-process worker thread [aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: ‘.\myproject.exe’ arguments: ‘exec “C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll”’ path: 'C:\inetpub\sites\www.myproject.com' [aspnetcorev2_inprocess.dll] Known dotnet.exe location: ‘’ [aspnetcorev2_inprocess.dll] Process path ‘.\myproject.exe’ is not dotnet, treating application as standalone or portable with bootstrapper [aspnetcorev2_inprocess.dll] Checking application.dll at ‘C:\inetpub\sites\www.myproject.com.\myproject.dll’ [aspnetcorev2_inprocess.dll] Checking hostfxr.dll at ‘C:\inetpub\sites\www.myproject.com.\hostfxr.dll’ [aspnetcorev2_inprocess.dll] hostfxr.dll found app local at ‘C:\inetpub\sites\www.myproject.com.\hostfxr.dll’, treating application as standalone [aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: ‘’ hostfxr path: ‘C:\inetpub\sites\www.myproject.com.\hostfxr.dll’ arguments: [aspnetcorev2_inprocess.dll] Argument[0] = ‘C:\inetpub\sites\www.myproject.com.\myproject.exe’ [aspnetcorev2_inprocess.dll] Argument[1] = ‘exec’ [aspnetcorev2_inprocess.dll] Argument[2] = ‘C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll’ [aspnetcorev2_inprocess.dll] In-process callbacks set [aspnetcorev2_inprocess.dll] Event Log: 'Application ‘C:\inetpub\sites\www.myproject.com' started the coreclr in-process successfully.’ End Event Log Message.

Attached is WER file I managed to find. Report.wer.log

We are using the temp bug fix for getting CurrentDirectory (see https://github.com/aspnet/AspNetCore/issues/4206#issuecomment-445612167) in Program.cs

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using myproject.Helpers;

namespace myproject
{
    public static class Program
    {
        public static void Main(string[] args)
        {
            CurrentDirectoryHelpers.SetCurrentDirectory(); // asp.net core 2.2 hack. In 3.0 this will be fixed: https://github.com/aspnet/AspNetCore/issues/4206#issuecomment-445612167
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
           WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>();
    }
}

Additional context

dotnet --info

.NET Core SDK (reflecting any global.json): Version: 2.2.100 Commit: b9f2fa0ca8

Runtime Environment: OS Name: Windows OS Version: 10.0.14393 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\2.2.100\

Host (useful for support): Version: 2.2.0 Commit: 1249f08fed

.NET Core SDKs installed: 2.1.300 [C:\Program Files\dotnet\sdk] 2.1.301 [C:\Program Files\dotnet\sdk] 2.2.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

[samcic]

Hi guys,

We’re jumping in here because we’re seeing a very similar issue. We too are running Asp.Net Core 2.2.1 using the in-process option and get a “Access violation - code c0000005” exception which kills our w3wp.exe process. We’re hosted on App Services on Azure.

We only started seeing these issues after upgrading to 2.2.x and switching to the In-Processs hosting model.

The crashes happens “rarely” (perhaps once or twice a day), but definitely enough to be taking up most of our attention because it causes restarts on our production servers.

Importantly, we managed to get a dump file (using the CrashDiag Azure extension) and can open it in Visual Studio locally which points the finger at a particular call in this framework. For security reasons I assume I shouldn’t give too many details here, just in case it might get exploited in some way.

We’d be happy to share the dump file with the team for analysis. We’re not really experts at looking at such files so we’d be glad if we got some assistance resolving this problem (and presumably you guys have a motivation to fix it).

Perhaps somebody could provide an email address to reach out to me about sharing the dump file and working together to fix this.

For now, we’ll try to switch back to Out-Of-Process (as suggested above by @jkotalik ) and see if things calm down on our side.

We look forward to hearing back from someone!

Thanks

[jkotalik]

@shapeh go ahead and share the dump with me to my email listed on my github profile.