ErrorCollector reports AssumptionViolatedException as failure when assumption is not meet multiple times.

Sample test:

import org.junit.Assume;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ErrorCollector;

public class SampleTest {

    @Rule
    public ErrorCollector errorCollector = new ErrorCollector();

    @Test
    public void test() throws Exception {
        errorCollector.checkSucceeds(() -> {
            doSth();
            return null;
        });
        errorCollector.checkSucceeds(() -> {
            doSth();
            return null;
        });
    }

    private void doSth() {
        Assume.assumeTrue(false);
    }

}

Expected result: test ignored. Actual: test failed.

Working as expected (ie. test results as ignored) when:

    @Test
    public void test() throws Exception {
        errorCollector.checkSucceeds(() -> {
            doSth();
            return null;
        });
    }

[kcooney]

After working on #1371 here’s what I think we should do

• If the Callable passed to ErrorCollector.checkSucceeds() throws an AssumptionViolatedException then checkSucceeds() should wrap it with an AssertionError (with a useful message) before calling addError()
• If an AssumptionViolatedException is passed to addError() then addError() should wrap it with an AssertionError (with a different message) before adding it to errors

I don’t like the idea of changing checkSucceeds() and/or addError() to throw exceptions if passed AssumptionViolatedException. Current users of these methods can reasonably expect that they will never throw an exception. In addition, if these methods did throw when encountering AssumptionViolatedException that would put a high burden on the callers. ErrorCollector rethrows any collected errors, so deferring the failure to verify() should be safe.

That being said, maybe addError() should throw a NullPointerException if you pass in a null (currently if you do this, MultipleFailureException.getMessage() will throw a NullPointerException). The caller shouldn’t have a problem avoiding passing in a null.

[kcooney]

I am actually now leaning to having ErrorCollector wrap AssumptionViolationException in another exception if it is the only collected exception. That way, if you have any collected errors you’ll get a failure (not a skipped test)