Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PyPy investigation notes

See original GitHub issue

I was curious how fast py-evm would run in pypy, and it sent me down a rabbit hole. This issue is just a way to note what I learned along the way.

pypy won’t work out of the box because of the dependency on pysha3 (via eth-bloom), which fails to compile under pypy
I ran eth-bloom tests as a rough speed proxy with several different configurations:
- pysha3: 0.5s
- pure python implementation in cpython: 25s
- pure python implementation in pypy: 9.5s
- pure python implementation in cython: 20.5s
- after a couple of type annotations in cython: 16.5s
- cython in pypy: crash at runtime with TypeError: object.__new__(builtin_function_or_method) is not safe, use builtin_function_or_method.__new__()
Optimizing sha3 in cython doesn’t matter if it doesn’t work under pypy (because we might as well keep the c extension version). Although I do expect that it would be possible to make it work under pypy with some tweaks.
Punting on pypy and working on a cython trie implementation (which was Piper’s first instinct) might be the most fruitful next step

Issue Analytics

State:
Created 6 years ago
Reactions:1
Comments:10 (8 by maintainers)

Top GitHub Comments

2reactions

carvercommented, Mar 4, 2018

From a basic local test, py-evm appears to be pypy3 compatible now 🎉

I converted the tox env py36-native-blockchain to pypy3-native-blockchain and ran both tests sequentially.

py36 ran in 131.4s
pypy3 ran in 70.8s

Which is pretty consistent with most other tests I’ve tried (eth-abi, web3.py, etc). Pypy3 seems to roughly double the speed of py36. 👍

Perhaps we could convert some/all of the fixture tests to run on pypy3-only. Unfortunately, they have been getting slower as more stuff gets implemented, and some of them are starting to fail regularly. (I think because of the 45min hard limit in travis boxes). The environments py3{5,6}-{native,rpc}-state-byzantium seem to fail a lot, for example.

My reasoning is: we’re testing for python compatibility in other core tests, and we don’t need the full matrix of python compatibility and blockchain protocol compatibility. We can just test python compatibility on core tests, and blockchain compatibility on pypy3.

Our CI situation would go from barely tolerable to unusable if we added pypy3 for every test, and this summer: py3.7.

2reactions

carvercommented, Feb 7, 2018

I believe the old issue blocking pysha3 got fixed, but there is another one: https://github.com/tiran/pysha3/issues/17 ie~ https://bitbucket.org/pypy/pypy/issues/2723/undefined-symbol-pyunicode_new-in-pypy3#comment-None

Anyway, the path @onyb started down seems like the best one for getting pypy live.

Top Results From Across the Web

Performance | PyPy

Note that a JIT scope is a run-time phenomenon, not a compile-time one. ... is not documented here, please report it to our...

A brief experiment with PyPy - LWN.net

One other interesting thing to note is that PyPy only made half as many system calls. That called for some investigation.

PyPy for low-latency systems

After a bit of investigation, we concluded that most (although not all) of the spikes were caused by the GC kicking in at...

Support for PyPy3 · Issue #17 · tiran/pysha3 - GitHub

Currently, the missing component is PyUnicode_New. I posted that issue to the pypy project: ... PyPy investigation notes ethereum/py-evm#138.

Python 3.8 status · Wiki · PyPy / pypy · GitLab

Some individual notes per test file. easy (eg error messages). test_typing; test_dictcomps; test_mmap; test_pyexpat; test_struct; test_cmd_line_script ...