Benchmark setup seems to be biased against async callback style libraries

I cloned bluebird to my machine and ran the benchmark tests to find how out my async callback style library(under development) stands up against the known async libraries. At first, the results were not surprising. Bluebird came out on top in parallel with the current bluebird setup. By changing the setup for callback style libraries(the way it should be, I think), the results were dramatically different. The extra scope lookups and memory allocations make a whole lot of difference.

The benchmark results from the original bluebird benchmark setup: -p 25 results for 10000 parallel executions, 1 ms per I/O op

file	time(ms)	memory(MB)
callbacks-baseline.js	261	63.12
promises-bluebird.js	421	106.31
callbacks-peterlu-asyncFlowF-parallel.js	469	93.04
callbacks-suguru03-neo-async-parallel.js	516	96.25
callbacks-caolan-async-parallel.js	842	157.09

The results from the new benchmark setup: -p 25 results for 10000 parallel executions, 1 ms per I/O op

file	time(ms)	memory(MB)
callbacks-baseline.js	255	62.99
callbacks-peterlu-asyncFlowF-parallel.js	285	73.48
callbacks-suguru03-neo-async-parallel.js	303	75.03
promises-bluebird.js	423	106.30
callbacks-caolan-async-parallel.js	611	139.36

Platform info: Linux 3.10.0-229.20.1.el7.x86_64 x64 Node.JS 5.0.0 V8 4.6.85.28 Intel® Core™ i3-4150 CPU @ 3.50GHz × 1

The changes I made to the setup:

New:

function fileInsertFor(i, tx) {
    var ins = FileVersion.insert({index: i})
    return function(callback) {
            ins.execWithin(tx, callback);
    };
}

Original:

function fileInsertFor(i, tx) {
    return function(callback) {
        FileVersion.insert({index: i}).execWithin(tx, callback);
    };
}

[peterluhub]

Yes, they do, but the big difference is that the promise version does not carry FileVersion.insert around like the callback style does. FileVersion.insert returns nothing but an object that has a few function pointers. The promise version executes FileVersion.insert before the queries array gets passed to Promise.all. So, the promise test run carries much less weight than the callback style version. And that is the key of the issue. The callback style test run does not need to execute FileVersion.insert later. It can execute FileVersion.insert just like the promise version. The same amount code is executed. The difference is where is that piece of code is executed.

The change I made does not execute less amount of code. It just changes where the code is executed. Therefore, it is still fair comparing to the promise version. The way you have it is definitely unfair to callback style test run.

[benjamingr]

I see no change in performance in Node v6.5.0 between the benchmarks with the change and without it. Maybe it was a temporary issue with v8.

That said, it does look like neo async performs a lot better on v6.5 (like in the chart OP posted) and we should update the benchmarks page to reflect that.