Provide a fast feedback loop for developing benchmarks

What steps will reproduce the problem?
1. Run example
2. Read console results


What is the expected output? What do you see instead?
Old version of caliper showed microbenchmark results in console.
New version seems to only show the number of tests run, etc.

What version of the product are you using? On what operating system?
 a3703ae4544b

Please provide any additional information below.

Original issue reported on code.google.com by agrothberg on 8 Apr 2013 at 10:04

[GoogleCodeExporter]

With all due respect… and I think you guys have done a great job on caliper 
so far… I think you have your head in the clouds over the web application.  
Pun half intended ;)

I mean at Google I'm sure everything is cloud cloud cloud but the world just 
isn't in that state yet… nor does it always NEED to be.

There's really nothing wrong with static compiled HTML and compiled output.  If 
the console doesn't do the job just put a BIG warning indication on it stating 
to view the web interface.  

A static HTML file would solve 99% of everyone's problems I think.

You would get your issues resolved with simple console output not showing 
enough detail and those of us behind firewalls or not wanting to rely on a 3rd 
party could get their reports.

Honestly this seems win/win for everyone :)

Original comment by [e-mail] on 5 Sep 2013 at 9:59

[GoogleCodeExporter]

If you are interested in manually analyzing the results, I've made my code for 
this public at https://code.google.com/p/caliper-analyze/

The statistics in my tool are carefully chosen (if you find any better 
statistic, please contribute!), in particular with respect to numerical 
precision (which is just too easy to do badly when computing variance; note 
that the statistics in previous caliper versions were flawed by this type of 
low-quality statistics: https://code.google.com/p/caliper/issues/detail?id=200 
as the webapp has not been released, I cannot see whether the code in there is 
more robust.)

Right now, my tool only does some basic console output. I've been working a 
little bit on predicting the scalability factors (i.e. linear, n log n, 
quadratic) by regression; but that is not working reliably yet. (c.f. 
https://code.google.com/p/caliper/issues/detail?id=119 )

Anyway, if you want a simple console-based way of analyzing your runs, this 
tool is probably useful for you. In particular, it will give you the properly 
weighted mean, and the properly weighted standard deviation, both absolute and 
relative.

The usual disclaimers about benchmarking of course still apply... in 
particular, if you see a high standard deviation, consider re-running the 
experiments with more iterations...

Original comment by [e-mail] on 1 Nov 2013 at 6:19