How best to specify that a Job should run quickly?

Hello,

I’ve just been using your FastAndDirtyConfig example from here: https://perfdotnet.github.io/BenchmarkDotNet/faq.htm

It doesn’t seem to do much unless I include Add(DefaultConfig.Instance):

public class FastAndDirtyConfig : ManualConfig
{
    public FastAndDirtyConfig()
    {
        Add(DefaultConfig.Instance); // *** add default loggers, reporters etc? ***

        Add(Job.Default
            .WithLaunchCount(1)     // benchmark process will be launched only once
            .WithIterationTime(100) // 100ms per iteration
            .WithWarmupCount(3)     // 3 warmup iteration
            .WithTargetCount(3)     // 3 target iteration
        );
    }
}

Do you think this should be part of the sample?

[jcansdale]

Unfortunately, it’s almost impossible to run it quickly and get a good accuracy at the same time in the general case. You just don’t know in advance what kind of problems you could get for a new benchmark.

I realize that BenchmarkDotNet has been designed to cater for a broad range of benchmarks - everything from a single IL operation to something longer running that might even hit a database. The kind of scenario I’m thinking of is developers who may want to turn some of their unit tests into more formalized benchmarks.

I’m hoping to find some parameters that will make it useful in the majority of cases. Fast enough so that people don’t mind running their benchmarks often, but not necessarily as accurate as they might want when running and keeping a history on their build server.

[AndreyAkinshin]

Unfortunately, it’s almost impossible to run it quickly and get a good accuracy at the same time in the general case. You just don’t know in advance what kind of problems you could get for a new benchmark. Right now you have the following ways:

• Use DryJob or ShortRunJob (a new brand for FastAndDirty) for a quick preview. These jobs are useful for verification (that you right a correct benchmark) and for getting first and very rough approximated results. I want also create a mutators for DryJob and ShortRunJob which can be easily applied for all user jobs.
• Use default job to get a good level of accuracy. (It takes some time)
• When you are playing with a benchmark for a long time and you are sure that there is no need to make a lot of iteration (because you have already seen many measurements), you can specify amount of iterations manually.

I totally understand that you want to have a magic way to quickly get a good results. But I still don’t think that it’s a good way when you are setting amount of iterations manually for each benchmark (or have a global “fast” config). There are always many questions in this case: why do you need exactly this number of iterations? Why your job has 10 iterations (not 9 or 11)?

The right question which you should ask is the following: what kind of trade-off between accuracy and benchmarking time do you want to achieve. Ideally, you should specify it on the API level. Probably, it will be a part of the Job.Accuracy. For example, now you can say: I want to get StandardError < 0.01 * Mean. It’s a reasonable accuracy requirement because you usually don’t care how much iterations will be performed (it’s more about implementation details), you care about the accuracy level.

I want to have a way to request a specific level of accuracy (like Low/Medium/High, it could be a magic enum or a set of accuracy presets). Unfortunately, it’s a really hard issue and it requests a lot of research. Also there is no a magic algorithm which will work fine for all benchmarks in the world. Hopefully, we can try to create a set of heuristics which works fine in 99% cases. =) Also I’m working on analyzers which can look at the current set of measurements and say something like “Hey, StdDev looks too big, probably your should do more iterations for this method”. But all of it is “work in progress”, I need much more time for that.