Benchmark not run in MS Test test class

Given the following test class in MS Test:

[TestClass]
public class TestBenchmarkDotNet
{
    [TestMethod]
    public void Test()
    {
        BenchmarkRunner.Run<TestBenchmarkDotNet>();
    }

    [Benchmark]
    public void Benchmark()
    {
        Thread.Sleep(5000);
    }
}

Running Test() from the Visual Studio test runner will not trigger the benchmark method at all, neither in Release or in Debug mode. Is this scenario supported?

[mms-]

@adamsitnik Agree 😃 Especially now that VS 2015 seems to be using just as much CPU as the unit test while it’s running.

But it still has merit. I’m referring to code refactoring with performance as a goal. Meaning you have old code and new code and want to compare what is faster, all the while you need to maintain correctness. Since tests will be updated anyway when this happens you might as well get feedback on performance.

That is often before one does a serious benchmark on a ready piece of code there is a period (usually long) of experimentation to see what gives better results, a guided search if you will. Accuracy of the benchmark does not matter here just the direction, is it faster or not. Getting this info from the IDE would increase productivity, weather through unit tests or some one other VS panel. Perhaps this is something the Visual Studio team could integrate.

[ig-sinicyn]

@adamsitnik We’ve integration with nUnit, xUnit and MsTest. xUnit one is quite lame (waiting for https://github.com/xunit/xunit/issues/908), others two work fine. I’m on vacation this week, so no new commits for some time, sorry 😃

About accuracy. As @mms wrote there’s actually no need in accuracy in unit tests. The thing you want is not accuracy but repeatability. Same code should produce same results on different runs, different machines and on different environments. If this is false the entire thing is failed. We have our tests running on appveyor CI, local buildserver, dev machines, ultrabooks and even tablets. No failures:)

* Of course, all above is true for “common” benchmarks, those that depend hardly on CPU features, memory available or on IO latency will require same-level hardware to get similar results.