Include match info when throwing a FakeItEasy.UserCallbackException: Argument matcher [...]

This is kind of a question rather then a bug.

I am wondering if it might be smarter to validate all rules, before throwing an exception that any one rule is not working. See comments below in the code example.

using System;
using FakeItEasy;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Internal;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace fm.Extensions.Logging.Tests
{
    [TestClass]
    public sealed class FakeItEasyBugTests
    {
        /// <summary>
        /// This test fails with
        /// Test method fm.Extensions.Logging.Tests.FakeItEasyBugTests.FailTest threw exception: 
        /// FakeItEasy.UserCallbackException: Argument matcher <ex => (ex.Message == "Test Exception")> threw an exception. See inner exception for details. ---> System.NullReferenceException: Object reference not set to an instance of an object.
        ///     at lambda_method(Closure , Exception )
        ///    at FakeItEasy.Core.DefaultArgumentConstraintManager`1.MatchesConstraint.FakeItEasy.Core.IArgumentConstraint.IsValid(Object argument) in C:\projects\fakeiteasy\src\FakeItEasy\Core\DefaultArgumentConstraintManager.cs:line 85
        /// --- End of inner exception stack trace ---
        ///     at FakeItEasy.Core.DefaultArgumentConstraintManager`1.MatchesConstraint.FakeItEasy.Core.IArgumentConstraint.IsValid(Object argument) in C:\projects\fakeiteasy\src\FakeItEasy\Core\DefaultArgumentConstraintManager.cs:line 89
        ///    at FakeItEasy.Expressions.ExpressionCallMatcher.ArgumentsMatchesArgumentConstraints(ArgumentCollection argumentCollection) in C:\projects\fakeiteasy\src\FakeItEasy\Expressions\ExpressionCallMatcher.cs:line 112
        ///    at FakeItEasy.Expressions.ExpressionCallMatcher.ArgumentsMatches(ArgumentCollection argumentCollection) in C:\projects\fakeiteasy\src\FakeItEasy\Expressions\ExpressionCallMatcher.cs:line 100
        ///    at FakeItEasy.Configuration.RuleBuilder.RuleMatcher.Matches(IFakeObjectCall call) in C:\projects\fakeiteasy\src\FakeItEasy\Configuration\RuleBuilder.cs:line 358
        ///    at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)
        ///    at FakeItEasy.Core.FakeAsserter.AssertWasCalled(Func`2 callPredicate, Action`1 callDescriber, CallCountConstraint callCountConstraint) in C:\projects\fakeiteasy\src\FakeItEasy\Core\FakeAsserter.cs:line 33
        ///    at FakeItEasy.Configuration.RuleBuilder.MustHaveHappened(CallCountConstraint callCountConstraint) in C:\projects\fakeiteasy\src\FakeItEasy\Configuration\RuleBuilder.cs:line 196
        /// </summary>
        [TestMethod]
        public void FailTest()
        {
            ILogger fake = A.Fake<ILogger>();

            fake.Trace("Test Message");
            fake.Trace(new Exception("Test Exception"), "Test Message");

            A.CallTo(() => fake.Log<object>(LogLevel.Trace, 0,
                A<FormattedLogValues>.That.Matches(f => f.ToString() == "Test Message"),
                A<Exception>.That.IsNull(),
                A<Func<object, Exception, string>>.Ignored)).MustHaveHappenedOnceExactly();

            A.CallTo(() => fake.Log<object>(LogLevel.Trace, 0,
                A<FormattedLogValues>.That.Matches(f => f.ToString() == "Test Message"),
                // #1: The following line will fail, because it is matched against the first log message, which has no exception.
                A<Exception>.That.Matches(ex => ex.Message == "Test Exception"),
                A<Func<object, Exception, string>>.Ignored)).MustHaveHappenedOnceExactly();

            A.CallTo(() => fake.Log<object>(LogLevel.Trace, 0,
                A<FormattedLogValues>.That.Matches(f => f.ToString() == "Test Message"),
                // #2: Workaround (check for null)
                A<Exception>.That.Matches(ex => ex != null && ex.Message == "Test Exception"),
                A<Func<object, Exception, string>>.Ignored)).MustHaveHappenedOnceExactly();
        }
    }
}

[blairconrad]

@Mertsch, I like the suggestion too, although I’d suggest some small changes.

1. I think it’d be slightly more clear if the message read “…when matching against call”. Maybe I worry too much.
2. I think the “1:” on the second line is superfluous, since we’ll only ever have one call in the list
3. In the example, ex is NULL, which reads very nicely when formatted inline as you’ve done. However, it may be something that formats into a large amount of data, complicating the message. It’s also going to be repeated down in the description of the call. I’d consider rewording the first line. Perhaps

   FakeItEasy.UserCallbackException: Argument matcher <ex => (ex.Message == "Test Exception")> threw an exception when matching against parameter "exception" in call
   

   (see how I snuck in my suggestion from line 1?)

Things to note:

1. sometimes the matcher will be a Func, not an Expression<Func>, so we won’t have a great description of the matcher. There’s not much we can do about that
2. sometimes the matcher will operate on the whole call, not individual arguments, so the message will need to be adapted for that case.
3. weak or faulty argument formatter functions (or ToString methods) on the function parameters will weaken the output. But that’s no different from the situation we have now

@Mertsch, since this issue bit you, are you interested in working on it?

[blairconrad]

Hey, @Mertsch. I’m sorry you had a bad experience.

My initial reaction is that suppressing the exceptions in user-supplied callbacks worries me. To my mind (and I know this may not be in the forefront of users’ minds when they’re writing them, but) user-supplied callbacks should be robust. If they throw on unexpected input, it feels like an indication of a potentially faulty callback. I’d be inclined to alert the user any time the callback threw. I think it’s better than having them just wonder why an assertion wasn’t used to match a call.

I’m disappointed that the new UserCallbackException wasn’t enough to help you quickly identify the problem. I’d’ve hoped knowing that a NullReferenceException was thrown by a callback supplied to a particular assertion would help the test-writer quickly zoom in on the cause, but it looks like we’re still falling short. Of course in your case, things are complicated by the fact that there are two callbacks in the one assertion: f => f.ToString() == "Test Message" and ex => ex.Message == "Test Exception", which would slow things down.

A more specific error message may help, but could be tricky. We could potentially output the name of the parameter that the callback failed on. What think you all?

something like A<Exception>.That.IsNotNull(ex => ex.Message == "Test Exception")

Honestly, I don’t think this helps enough (or really at all). It’s functionally the same as A<Exception>.That.Matches(ex => ex != null && ex.Message == "Test Exception") (or even A<Exception>.That.Matches(ex => ex?.Message == "Test Exception")), and I don’t think it’s worth the change, especially given that users would still have to consider that the Exception might be null and use an alternative to Matches.