Benchmark setup and caching of repeated operations

I’m developing a library where an operation might be performed once or multiple times. I’d like to benchmark how long setup takes as well as how much caching might speed up subsequent operations.

For example, say I wanted to investigate the performance of compiled vs static regular expressions.

What I’d like is to be able to do the following:

        [Benchmark]
        public IEnumerable<Action> Regex_Match_Compiled()
        {
            string value;
            var regex = new Regex("[ ](.*)!", RegexOptions.Compiled);
            for (int i = 0; i < 3; i++)
            {
                yield return () => value = regex.Match("Hello, World!").Groups[1].Value;
            }
        }

        [Benchmark]
        public IEnumerable<Action> Regex_Match_Static()
        {
            string value;
            for (int i = 0; i < 3; i++)
            {
                yield return () => value = Regex.Match("Hello, World!", "[ ](.*)!").Groups[1].Value;
            }
        }

Method	Median	StdDev
Regex_Match_Compiled(1)	1,035,622.7664 ns	21,349.3871 ns
Regex_Match_Compiled(2)	10234.7148 ns	60,138.4160 ns
Regex_Match_Compiled(3)	10640.9544 ns	21,160.7904 ns
Regex_Match_Static(1)	943.3168 ns	29.4935 ns
Regex_Match_Static(2)	948.445 ns	67.2509 ns
Regex_Match_Static(3)	798.6434 ns	19.6410 ns

At the moment, I could do something like this:

        [Benchmark]
        public void Regex_Match_Compiled_1()
        {
            Execute(Regex_Match_Compiled(1));
        }

        [Benchmark]
        public void Regex_Match_Compiled_2()
        {
            Execute(Regex_Match_Compiled(2));
        }

        [Benchmark]
        public void Regex_Match_Compiled_3()
        {
            Execute(Regex_Match_Compiled(3));
        }

        [Benchmark]
        public void Regex_Match_Static_1()
        {
            Execute(Regex_Match_Static(1));
        }

        [Benchmark]
        public void Regex_Match_Static_2()
        {
            Execute(Regex_Match_Static(2));
        }

        [Benchmark]
        public void Regex_Match_Static_3()
        {
            Execute(Regex_Match_Static(3));
        }

        static void Execute(IEnumerable<Action> actions)
        {
            foreach(var action in actions)
            {
                action();
            }
        }

        public IEnumerable<Action> Regex_Match_Compiled(int n)
        {
            string value;
            var regex = new Regex("[ ](.*)!", RegexOptions.Compiled);
            for (int i = 0; i < n; i++)
            {
                yield return () => value = regex.Match("Hello, World!").Groups[1].Value;
            }
        }

        public IEnumerable<Action> Regex_Match_Static(int n)
        {
            string value;
            for (int i = 0; i < n; i++)
            {
                yield return () => value = Regex.Match("Hello, World!", "[ ](.*)!").Groups[1].Value;
            }
        }

Method	Median	StdDev
Regex_Match_Compiled_1	1,035,622.7664 ns	21,349.3871 ns
Regex_Match_Compiled_2	1,045,857.4812 ns	60,138.4160 ns
Regex_Match_Compiled_3	1,056,498.4356 ns	21,160.7904 ns
Regex_Match_Static_1	943.3168 ns	29.4935 ns
Regex_Match_Static_2	1,891.7618 ns	67.2509 ns
Regex_Match_Static_3	2,690.4052 ns	19.6410 ns

It’s not pretty and I have to subtract the results to see how long a single operation takes.

Does that make sense or is there already some way to do this?

[adamsitnik]

I think my confusion has been because [Setup] doesn’t behave as someone who is used to NUnit would expect. It isn’t called before each [Benchmark] method, but is a one-time-setup that is called even before the warm-up iterations. This means that any caching done by the [Benchmark] methods is lost during warm-up.

@jcansdale I have just added extra explanation to Setup and Cleanup attribute summary to avoid confusion in the future. Thanks for the input!

[jcansdale]

It looks like the regex is parsed and the source gets prepared when the Regex is constructed. When Match gets called it looks in a cache for a compiled assembly (keyed off the source + parameters). Compilation only happens if it can’t find anything in the cache. That’s what I think is going on - the Regex source is pretty hairy. 😉