sortWith() and orderWith() should respect @FixMethodOrder

When you order a suite using Request.sortWith() all of the runners that implement Sortable (like BlockJUnit4ClassRunner) will reorder their methods using the passed-in Comparator, even if the ordering was specified with @FixMethodOrder. The same holds true for Request.orderWith(). JUnit should instead not reorder methods for classes annotated with @FixMethodOrder

Background:

Someone at my company was trying to implement code to shuffle method orders to catch tests that produced different pass/fail results based on ordering (these shuffling tests are useful, for instance, if you want to ensure that you could always add @Ignore to a broken test without causing otherwise-passing tests to start failing). We have a very large code base, and some tests are annotated with @FixMethodOrder because we know that they only pass if run in a specific order. We have enough of these that it will take a significant amount of time to fix them, especially for parts of the code where the original authors have left the company.

We had to write custom shuffling code, because the code in JUnit for ordering did not respect @FixMethodOrder. In some cases, our customized shuffling code has to bail out with a “Shuffling not supported for this suite” error message.

Can we please have ParentRunner.sort(Sorter) and ParentRunner.order(Orderer) do nothing if the Description indicates that the class was annotated with @FixMethodOrder?

I do realize that this would be a change in behavior from previous versions of JUnit. I don’t think the change I’m proposing would cause tests to fail. The only way that could happen is if there is some code outside of JUnit that specifically orders the suite in an order, but if projects run into this problem when trying to migrate to 4.13, the easy work-around would be to remove the @FixMethodOrder annotation.

[marcphilipp]

Yes, I think so unless there’s a really good reason.

Since orderWith() is new, I think we should make the change now and postpone 4.13.

[kcooney]

@marcphilipp while Request,orderWith() and @OrderWith are new, Request.sortWith() is not. I think it might be odd to have the orderWith() respect @FixMethodOrder but sortWith() not respect it. That being said, if we change the sorting code so sortWith() respects @FixMethodOrder we would likely need to see who might broken by the change, which would require a delay of the 4.13 release.

I’ll start looking into where we could apply the fixes.