Add benchmarking test suite.

Currently we don’t have good benchmarking tools.

Lets write a script that does the following.

• For each set of mainnet VM rules, build a chain of X empty blocks. ✅
  • one using mine_block ✅
  • another using import_block ✅
• For each set of mainnet VM rules, build a chain of X block, each full of simple value transfer transactions. ✅
  • sender should be static across all tests ✅
  • one run using empty to accounts. ✅
  • one run using existing to accounts. ✅
• For each set of mainnet VM rules, benchmark using the open zeppelin ERC20 token contract.
  • one run with X calls to transfer
  • one run with X calls to approve
  • one run with X calls to transferFrom
• ~For each set of mainnet VM rules, benchmark deployment of the open zeppelin ERC20 token contract.~ ✅
• For each set of mainnet VM rules, benchmark DOS style transaction spam.
  • Make X calls to a contract which uses all available gas to create empty contracts.
  • Make X calls to a contract which uses all available gas on SSTORE operations.
  • Two more same as above but which revert at the end of contract execution.
• ~For each set of mainnet VM rules, benchmark gas per second for block import.~ ✅
  • ~Measure for pure value transfer transactions.~ ✅
  • ~Measure for representative EVM execution (which will have to be an approximation. probably worth starting with just ERC20 contracts and transfers and then expanding from there.~ ✅
• ensure blocks use pre generated fixtures with real blocks that qualify the PoW so that we don’t miss to benchmark the POW validation/verification overhead
• have at least one benchmark to use LevelDB rather than MemoryDB (ideally all of them but may be too big of an performance hit)

These benchmark scripts should produce detailed output to the console and be added as a stand-alone test run in our CI environment. Any failing benchmark scripts should cause CI to fail.

Stats worth tracking:

• time to run
• total DB size

[pipermerriam]

That is some good looking 💩

[cburgdorf]

@pipermerriam I started hacking something together but it’s really just a dirty pile of 💩 at this point. Just opening the PR to for the illusion of progress 😉

https://github.com/ethereum/py-evm/pull/650