Comparison to previous results

Does benny have any mechanism to compare this benchmark to my last round, so that I can say something of the form “in this release the library is on average 2% faster” ?

[caderek]

I’m planning to add this, but it is not ready yet. In the meanwhile you can use this script to diff two versions of the JSON results:

/* diff.js */
const fs = require('fs')

const fileA = JSON.parse(fs.readFileSync(process.argv[2]).toString())
const fileB = JSON.parse(fs.readFileSync(process.argv[3]).toString())

const diffs = fileB.results.map((resultB) => {
  const oldResult = fileA.results.find(
    (resultA) => resultA.name === resultB.name,
  )

  if (!oldResult) {
    return { name: resultB.name, diff: null }
  }

  const diff = ((resultB.ops - oldResult.ops) / oldResult.ops) * 100

  return {
    name: resultB.name,
    diff,
  }
})

console.log(
  diffs
    .map(
      ({ name, diff }) =>
        `${name}: ${diff.toFixed(2)}% ${
          diff > 0 ? 'faster' : diff < 0 ? 'slower' : 'same'
        }`,
    )
    .join('\n'),
)

const changed = diffs.filter((item) => item.diff !== null)
const average = changed.reduce((a, b) => a + b.diff, 0) / changed.length

console.log(
  `Average: ${average.toFixed(2)}% ${
    average > 0 ? 'faster' : average < 0 ? 'slower' : 'same'
  }`,
)

Usage:

node diff.js path/to/oldBench.json path/to/newBench.json

Example output:

Case one: 0.22% faster
Other case: -2.50% slower
Average: -1.14% slower

https://gist.github.com/caderek/93394875730cc1b9f6750e527e7b9fcf

Hope this will help!

I will leave this issue open as a feature request.

[StoneCypher]

Please don’t paint issues. Make a new one instead