question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Benchmarks are incorrect

See original GitHub issue

You’re using GlobalSetup/Cleanup instead of IterationSetup/Cleanup to create the Publisher, so after the first few runs TryEnqueue just starts returning false without doing any work.

Once you fix that, you’ll start getting warnings that the op count is too low to get legitimately valid results.

Once everything is fixed, you’ll probably get something like this, if you use a message size of 128, queue capacity of 33554432 bytes, and 246723 operations per iteration. (e.g. [Benchmark(Description = "Enqueue messages", OperationsPerInvoke = 246723)] and for (var i = 0; i < 246723; ++i))

Of course, per machine, YMMV…

|                                                    Method |     Mean |    Error |   StdDev |  Gen 0 | Allocated |
|---------------------------------------------------------- |---------:|---------:|---------:|-------:|----------:|
|                                        'Enqueue messages' | 543.8 ns |  4.19 ns |  3.50 ns |      - |         - |
|                            'Enqueue messages (zero-copy)' | 547.2 ns |  2.78 ns |  2.32 ns |      - |         - |
|             'Enqueue messages (zero-copy, func pointers)' | 557.0 ns |  3.71 ns |  3.29 ns |      - |         - |
|      'Enqueue and dequeue messages (buffered allocating)' | 724.7 ns |  5.08 ns |  4.50 ns | 0.0284 |     152 B |
|             'Enqueue and dequeue messages (buffer reuse)' | 712.3 ns |  6.32 ns |  5.27 ns |      - |         - |
|                'Enqueue and dequeue messages (zero-copy)' | 709.4 ns | 13.67 ns | 15.20 ns |      - |         - |
| 'Enqueue and dequeue messages (zero-copy, func pointers)' | 718.9 ns |  7.67 ns |  6.80 ns |      - |         - |

You don’t have the zero-copy ops yet, but you can review a future PR at https://github.com/StirlingLabs/interprocess

It’s still quite fast, but 7ns/op is 28 CPU cycles/op at 4ghz (between 7 and 36 GP instructions), which is a dead giveaway that no work was being done; it’s literally impossibly fast, it has to just be checking a value and returning false. Putting the benchmark into debug mode (BenchmarkRunner.Run(typeof(Program).Assembly, new DebugInProcessConfig());) and stepping into it confirmed it.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
prezaeicommented, Oct 3, 2021

I updated the enqueue message benchmark test code. I will update the README with the latest numbers soon. Thanks for reporting it @TYoungSL.

0reactions
prezaeicommented, Sep 19, 2021

Hey, happy to take ideas/PRs in small chunks. Can you do a short design write-up/proposal on each change first? We can review and get them in. I need to be careful here because this is heavily used at Microsoft for many of our services.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Benchmark Error Definition
Benchmark error is a situation in which the wrong benchmark is selected in a financial model, causing the model to produce inaccurate results....
Read more >
The Biggest Benchmarking Mistakes And Pitfalls You Must ...
Lack of context. In most cases, the benchmark data being compared is stripped of its short-term and long-term context, meaning it tells you...
Read more >
Beware of Benchmarking: Bad Data is Worse Than No Data
Benchmarking has been popular for at least twenty years. However, if the wrong items are benchmarked or the result is inaccurate, bad data...
Read more >
The Most Common Benchmarking Mistakes
Outdated data can give you a false sense of how your company or product is performing. Focusing on the wrong metrics: It is...
Read more >
The Problem With Benchmarks
Benchmarks long have been used to compare products, but what makes a good benchmark and who should be trusted with their creation?
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found