Testing the new resolver (test suite, CI)

Now that we have at least the start of the new resolver in place, we need to consider how we integrate it into the test suite. I’ve modified one of the very basic install tests to run twice, once with the legacy resolver and once with --unstable-feature=resolver, but this obviously isn’t a scalable approach.

Some issues we therefore need to consider:

1. How do we have a more general mechanism that allows us to run tests with both old and new resolvers?
2. How do we handle tests that use feaures the new resolver doesn’t have implemented yet (effectively xfail for the new resolver, but not for the current one)?
3. How do we cater for tests where we expect different results? (Tests that rely on quirks of the legacy resolver, explicit tests of new resolver functionality, …)

Ultimately, we want the full test suite to be run for both resolvers, so that we have a good regression test ensuring that we have feature parity, but we’re likely a good way from that position yet. But the only way to know for sure how much there still is to do is to (reasonably regularly) run the full suite with the new resolver.

[pfmoore]

@pradyunsg Can you take a look at my comment on #8462? I swear I tried to pick a simple one, and it turned complicated. I think I’m jinxed… 🙄

[pradyunsg]

I’d say before. 😃

I’m considering that a part of #8937.