Performance Optimizations

What is wrong?

There is rudamentary full sync code in place and with that, we’re finding performance issues.

• https://drive.google.com/open?id=1OwLcvS-ONGmgucL4Tccf5nqP2r_QCglv
• github-mirror

The link above is the cProfile output which can be loaded using the pstats standard library module to see where time is spent.

Known “low hanging fruit”

The following things are where we’re seeing the most time spent which can likely be optimized.

• Remove support for python 2 from pyrlp
• Further pyrlp optimizations
• Caching of the VMState.prev_hashes generation.

Ways to make things faster

Cython looks to be a very promising way for us to do certain optimizations. We should spend a little time doing discovery to see how this well this works and what kind of performance gains we can get.

[gsalgado]

Fresh cProfile stats, now with all performance improvements and a much smaller DB (thanks to #600): block-import-stats.txt

TL;DR: trie seems to be our bottleneck now, together with some ChainDB methods, but some of the latter can probably be cached

[pipermerriam]

We may be talking about the same thing.

https://github.com/ethereum/py-evm/blob/master/evm/db/backends/base.py

I think this needs a new API: BaseDB.batch_set(...)

MemoryDB would implement this as a loop calling down to individual set calls. The LevelDB backend would use the batch APIs from plyvel.