Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Debug.Assert cause test host process to crash

See original GitHub issue

Description

Debug.Assert cause test host process to crash when tests target .NET Core.

Not sure if this is the right place to report this, but starting from here as there was already an issue here about it: #1022

Steps to reproduce

  • Create a new xUnit project (C#) targeting netcoreapp2.1 or netcoreapp3.0 or netcoreapp3.1
  • Create a test using Debug.Assert
    [Fact]
public void Test1()
{
    Debug.Assert(false);
}

Expected behavior

Ideally the test should just fail in my opinion. Otherwise it should just have the previous behaviour on .NET Framework: you get Debug.Assert window

image

Actual behavior

❯ dotnet test
Test run for C:\dev\pub\alefranz\DebugAssertTests\DebugAssertTests\bin\Debug\netcoreapp3.1\DebugAssertTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Process terminated. Assertion failed.
   at DebugAssertTests.UnitTest1.Test1() in C:\dev\pub\alefranz\DebugAssertTests\DebugAssertTests\UnitTest1.cs:line 12     at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Xunit.Sdk.TestInvoker`1.CallTestMethod(Object testClassInstance) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 150
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 257
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<InvokeTestMethodAsync>b__1()
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction)
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<InvokeTestMethodAsync>b__0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 242
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code)
   at Xunit.Sdk.TestInvoker`1.InvokeTestMethodAsync(Object testClassInstance) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 241
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestInvoker`1.InvokeTestMethodAsync(Object testClassInstance)
   at Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(Object testClassInstance) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestInvoker.cs:line 112
   at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 206
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0()
   at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code)
   at Xunit.Sdk.TestInvoker`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 189
   at Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(ExceptionAggregator aggregator) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs:line 84
   at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(ExceptionAggregator aggregator) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs:line 67
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(ExceptionAggregator aggregator)
   at Xunit.Sdk.TestRunner`1.<>c__DisplayClass43_0.<RunAsync>b__0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs:line 149
   at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code)
   at Xunit.Sdk.TestRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs:line 149
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestRunner`1.RunAsync()
   at Xunit.Sdk.XunitTestCaseRunner.RunTestAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCaseRunner.cs:line 139
   at Xunit.Sdk.TestCaseRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCaseRunner.cs:line 82
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestCaseRunner`1.RunAsync()
   at Xunit.Sdk.XunitTestCase.RunAsync(IMessageSink diagnosticMessageSink, IMessageBus messageBus, Object[] constructorArguments, ExceptionAggregator aggregator, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\XunitTestCase.cs:line 162
   at Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(IXunitTestCase testCase) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestMethodRunner.cs:line 45
   at Xunit.Sdk.TestMethodRunner`1.RunTestCasesAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs:line 136
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestMethodRunner`1.RunTestCasesAsync()
   at Xunit.Sdk.TestMethodRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs:line 106
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestMethodRunner`1.RunAsync()
   at Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(ITestMethod testMethod, IReflectionMethodInfo method, IEnumerable`1 testCases, Object[] constructorArguments) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestClassRunner.cs:line 168
   at Xunit.Sdk.TestClassRunner`1.RunTestMethodsAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs:line 213
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestClassRunner`1.RunTestMethodsAsync()
   at Xunit.Sdk.TestClassRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs:line 171
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestClassRunner`1.RunAsync()
   at Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(ITestClass testClass, IReflectionTypeInfo class, IEnumerable`1 testCases) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCollectionRunner.cs:line 158       at Xunit.Sdk.TestCollectionRunner`1.RunTestClassesAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCollectionRunner.cs:line 130
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestCollectionRunner`1.RunTestClassesAsync()
   at Xunit.Sdk.TestCollectionRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCollectionRunner.cs:line 101
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Xunit.Sdk.TestCollectionRunner`1.RunAsync()
   at Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(IMessageBus messageBus, ITestCollection testCollection, IEnumerable`1 testCases, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestAssemblyRunner.cs:line 235
   at Xunit.Sdk.XunitTestAssemblyRunner.<>c__DisplayClass14_2.<RunTestCollectionsAsync>b__2() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\Xuni

Test Run Aborted.

Environment

tested on dotnet SDK 3.1.101 and .NET 5 tested with Microsoft.NET.Test.Sdk between 16.2.0 and 16.5.0-preview-20200116-01

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:3
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
alefranzcommented, Feb 24, 2020

That was quick! Thanks @nohwnd !

1reaction
nohwndcommented, Feb 24, 2020

It is possible to plugin a custom trace listener to wrap the debug failure. I don’t think we can (or should) give you the pop up back. Instead we should translate the assert failure to an exception that will fail the test and will get reported the normal way.

Even more, because we have access to the stack trace we can look if there is source code available and even print the actual line of code that failed, imho this is very useful when you are looking at the logs (but not sure if it will be safe to do to make it to the official solution):

The below implementation will throw also on Debug.WriteLine. see #2335 for proper solution if you want to create your own listener.

  Test method UnitTestProject1.UnitTest1.FailedAssert threw exception:
UnitTestProject1.MyTraceListener+DebugAssertException: Debug failed at:
Debug.Assert(thisShouldBe10 == 10); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
  Stack Trace:
      at UnitTestProject1.MyTraceListener.WriteLine(String message) in /mnt/c/Projects/temp/ConsoleApp5/UnitTestProject1/UnitTest1.cs:line 47
   at System.Diagnostics.TraceListener.Fail(String message, String detailMessage)
   at System.Diagnostics.TraceInternal.Fail(String message, String detailMessage)
   at System.Diagnostics.TraceInternal.TraceProvider.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Assert(Boolean condition)
   at ConsoleApp5.Class1.MyMethod2() in /mnt/c/Projects/temp/ConsoleApp5/ConsoleApp5/Program.cs:line 38
   at ConsoleApp5.Class1.MyMethod1() in /mnt/c/Projects/temp/ConsoleApp5/ConsoleApp5/Program.cs:line 31
... shortened

Here is quick prototype code for inspiration. There are no guarantees but it should be easy to plugin into your solution before we fix this, because it does not need any help from the test framework. 😊

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Diagnostics;
using System.IO;
using System.Reflection;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [ClassInitialize]
        public static void SetupListeners(TestContext c)
        {
            Trace.Listeners.RemoveAt(0);
            Trace.Listeners.Add(new MyTraceListener());
        }

        [TestMethod()]
        public void FailedAssert()
        {
            var class1 = new Class1();

            class1.MyMethod1();
        }

        [TestMethod()]
        public void FailedFail()
        {
            var class1 = new Class1();

            class1.MyMethod3();
        }
    }

    class MyTraceListener : TraceListener
    {
        public override void Write(string message)
        {
            var stack = new StackTrace(true);
            throw new DebugAssertException(GetReport(message, stack), stack);
        }

        public override void WriteLine(string message)
        {
            var stack = new StackTrace(true);
            throw new DebugAssertException(GetReport(message, stack), stack);
        }

        private string GetReport(string message, StackTrace stack)
        {
            var frames = stack.GetFrames();
            string report = null;
            MethodBase debugMethod = null;
            StackFrame frame = null;
            foreach (var f in frames)
            {
                if (debugMethod != null)
                {
                    if (f.HasMethod())
                    {
                        frame = f;
                        break;
                    }
                }

                if (f.HasMethod())
                {
                    var m = f.GetMethod();
                    if (m.DeclaringType == typeof(Debug))
                    {
                        debugMethod = m;
                    }
                }
            }

            if (frame != null)
            {
                if (frame.HasSource())
                {
                    var fileName = frame.GetFileName();
                    var lineNumber = frame.GetFileLineNumber();
                    string line = null;
                    using (var sr = new StreamReader(fileName))
                    {
                        for (int i = 1; i < lineNumber; i++)
                            sr.ReadLine();
                        line = sr.ReadLine();
                    }

                    var filteredMessage = message == "Fail:" ? null : message;

                    report += Environment.NewLine;
                    report += Environment.NewLine; 

                    if (filteredMessage != null)
                    {
                        report += filteredMessage;
                    }

                    if (line != null)
                    {
                        report = string.Join(Environment.NewLine, "Debug failed at:", line.Trim());
                    }
                }
            }

            return report;
        }

        internal class DebugAssertException : Exception
        {

            public DebugAssertException(string message, StackTrace stack) : base(message ?? "Debug.Assert failed.")
            {
                StackTrace = stack.ToString();
            }

            public override string StackTrace { get; }
        }
    }

    public class Class1
    {
        public void MyMethod1()
        {
            MyMethod2();
        }

        public void MyMethod2()
        {
            var thisShouldBe10 = 20;

            Debug.Assert(thisShouldBe10 == 10);
        }

        public void MyMethod3()
        {
            Debug.Fail("this fails");
        }
    }
}

dotnet test
Test run for /mnt/c/Projects/temp/ConsoleApp5/UnitTestProject1/bin/Debug/netcoreapp3.1/UnitTestProject1.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
  X FailedAssert [64ms]
  Error Message:
   Test method UnitTestProject1.UnitTest1.FailedAssert threw exception:
UnitTestProject1.MyTraceListener+DebugAssertException: Debug failed at:
Debug.Assert(thisShouldBe10 == 10);
  Stack Trace:
      at UnitTestProject1.MyTraceListener.WriteLine(String message) in /mnt/c/Projects/temp/ConsoleApp5/UnitTestProject1/UnitTest1.cs:line 47
   at System.Diagnostics.TraceListener.Fail(String message, String detailMessage)
   at System.Diagnostics.TraceInternal.Fail(String message, String detailMessage)
   at System.Diagnostics.TraceInternal.TraceProvider.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Assert(Boolean condition)
   at ConsoleApp5.Class1.MyMethod2() in /mnt/c/Projects/temp/ConsoleApp5/ConsoleApp5/Program.cs:line 38
   at ConsoleApp5.Class1.MyMethod1() in /mnt/c/Projects/temp/ConsoleApp5/ConsoleApp5/Program.cs:line 31
   at UnitTestProject1.UnitTest1.FailedAssert() in /mnt/c/Projects/temp/ConsoleApp5/UnitTestProject1/UnitTest1.cs:line 25
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
.... shortened

  X FailedFail [5ms]
  Error Message:
   Test method UnitTestProject1.UnitTest1.FailedFail threw exception:
UnitTestProject1.MyTraceListener+DebugAssertException: Debug failed at:
Debug.Fail("this fails");
  Stack Trace:
      at UnitTestProject1.MyTraceListener.WriteLine(String message) in /mnt/c/Projects/temp/ConsoleApp5/UnitTestProject1/UnitTest1.cs:line 47
   at System.Diagnostics.TraceListener.Fail(String message, String detailMessage)
   at System.Diagnostics.TraceInternal.Fail(String message, String detailMessage)
   at System.Diagnostics.TraceInternal.TraceProvider.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Fail(String message)
   at ConsoleApp5.Class1.MyMethod3() in /mnt/c/Projects/temp/ConsoleApp5/ConsoleApp5/Program.cs:line 43
   at UnitTestProject1.UnitTest1.FailedFail() in /mnt/c/Projects/temp/ConsoleApp5/UnitTestProject1/UnitTest1.cs:line 33
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, 
... shortened


Test Run Failed.
Total tests: 2
     Failed: 2
 Total time: 1.3861 Seconds
Read more comments on GitHub >

github_iconTop Results From Across the Web

Using Debug.Assert may crash test run with a mangled ...
The exception that terminates the process when a Debug.Assert fails is mangled. It is not clear why the test run fails.
Read more >
How do I determine what exception caused Test Explorer ...
The test runner was running out of virtual memory due to running under 32 bit. But there is still at least one more...
Read more >
Debug.Assert Method (System.Diagnostics)
method is used to identify logic errors during program development. Assert evaluates the condition. If the result is false , it sends a...
Read more >
Debugging with assert() - a free Hacking with Swift tutorial
One level up from print() are assertions, which are debug-only checks that will force your app to crash if a specific condition isn't...
Read more >
Stop requiring only one assertion per unit test
Assertion Roulette doesn't mean that multiple assertions are bad. When I coach teams or individual developers in test-driven development ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Solution with netcoreapp3 and net472 projects

Next page

Enabling code coverage leads to errors in child processes