Kestrel server stops responding during test run using 3.0 previews 4 & 5

We’re upgrading an existing ASP.NET Core 2.2 application to 3.0 by maintaining a parallel PR/branch using the 3.0 previews to test and update things in advance of the RTM later this year.

As part of this web application, in our “integration” tests we self-host the application using Kestrel and drive browser automation headless UI tests using web driver and Selenium to test the application and UI behaviour.

One of the test jobs for this application uses BrowserStack Automate to validate that the UI works correctly when served to browsers we can’t otherwise easily test in our CI infrastructure, such as iPhone and macOS with Safari.

Since updating to preview 4 of 3.0, this test leg has stopped working and is exhibiting the behaviour of seemingly locking up and not responding. 6 of the 8 tests in the test suite pass successfully, but then the job hangs and eventually the test jobs times out after 30 minutes. The hang is roughly starting 6 minutes into the test run.

Using previews 2 & 3 the tests were behaviour as expected, and our master branch using 2.2 (2.2.4 right now) is also fine.

The interesting artifact from routing the logs to xunit output are these messages there’s a portion of below. These messages are output continuously until the test job is terminated at the 30 minute job timeout.

[17:46:57][dotnet test] [00:46:56 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:46:54 +00:00".
[17:46:57][dotnet test] [00:46:56 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:46:55 +00:00".
[17:46:57][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:46:57][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:46:54 +00:00".
[17:46:57][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:46:57][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:46:55 +00:00".
[17:47:07][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:07][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:06 +00:00".
[17:47:07][dotnet test] [00:47:06 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:03 +00:00".
[17:47:07][dotnet test] [00:47:06 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:06 +00:00".
[17:47:07][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:07][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:03 +00:00".
[17:47:18][dotnet test] [00:47:17 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:15 +00:00".
[17:47:18][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:18][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:14 +00:00".
[17:47:18][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:18][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:15 +00:00".
[17:47:18][dotnet test] [00:47:17 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:14 +00:00".
[17:47:18][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:18][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:16 +00:00".
[17:47:18][dotnet test] [00:47:17 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:16 +00:00".
[17:47:30][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:30][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:29 +00:00".
[17:47:30][dotnet test] [00:47:29 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:29 +00:00".
[17:47:31][dotnet test] [00:47:30 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:30 +00:00".
[17:47:31][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:31][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:30 +00:00".
[17:47:32][dotnet test] [00:47:31 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:31 +00:00".
[17:47:32][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:32][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:31 +00:00".
[17:47:33][dotnet test] [00:47:32 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:32 +00:00".
[17:47:33][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:33][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:32 +00:00".
[17:47:34][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:34][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:33 +00:00".
[17:47:34][dotnet test] [00:47:33 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:33 +00:00".
[17:47:34][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:34][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:28 +00:00".
[17:47:34][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:34][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:25 +00:00".
[17:47:34][dotnet test] [00:47:33 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:28 +00:00".
[17:47:34][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:34][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:27 +00:00".
[17:47:34][dotnet test] [00:47:33 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:25 +00:00".
[17:47:35][dotnet test] [00:47:33 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:26 +00:00".
[17:47:35][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:35][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:26 +00:00".
[17:47:35][dotnet test] [00:47:33 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:27 +00:00".
[17:47:45][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:45][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:44 +00:00".
[17:47:45][dotnet test] [00:47:44 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:44 +00:00".
[17:47:46][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:46][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:45 +00:00".
[17:47:50][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:50][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:47 +00:00".
[17:47:50][dotnet test] [00:47:49 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:47 +00:00".
[17:47:50][dotnet test] [00:47:49 WRN] Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:45 +00:00".
[17:47:50][dotnet test] warn: Microsoft.AspNetCore.Server.Kestrel[22]
[17:47:50][dotnet test]       Heartbeat took longer than "00:00:01" at "05/07/2019 00:47:46 +00:00".

Other test legs using the same approach, but without BrowserStack Automate in the mix, are working as expected with 3.0 preview 5.

Sorry that I can’t attach an easy repro as this is part of a moderately sized internal line-of-business application. Hopefully based on the description and further information you can ask me for to see if I can provide, we can get to the bottom of what’s going on and resolve.

/cc @bradygaster

[martincostello]

@anurse I can confirm that I can’t repro this with 3.1.1 with the linked repro project, or the original project where I found this issue. Closing.

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.