Generic Modules/Utils

Before moving on to the rest of libp2p and py-ipfs, we should look at the required generic utilities and determine which existing implementation we want to use, or if there isn’t one, who wants to work on it. Here is what we need (see https://github.com/ipfs/py-ipfs/issues/21#issuecomment-158019175 for more info):

• PeerID
  • js impl: https://github.com/diasdavid/js-peer-id
• PeerInfo
  • js impl: https://github.com/diasdavid/js-peer-info
• multihash
  • specs: https://github.com/jbenet/multihash
  • go impl: https://github.com/jbenet/go-multihash
  • js impl: https://github.com/jbenet/js-multihash
  • py impl: https://github.com/ivilata/pymultihash
• multihashing (optional, pymultihash delegates this mostly to hashlib)
  • specs: ???
  • js impl: https://github.com/jbenet/js-multihashing
• multiaddr
  • specs: https://github.com/jbenet/multiaddr
  • go impl: https://github.com/jbenet/go-multiaddr
  • js impl: https://github.com/jbenet/js-multiaddr
  • py impl: https://github.com/sbuss/py-multiaddr
• multistream
  • specs: https://github.com/jbenet/multistream
  • js impl: https://github.com/diasdavid/js-multistream
• multicodec
  • specs: https://github.com/jbenet/multicodec
  • go impl: https://github.com/jbenet/go-multicodec
  • py impl: https://github.com/multiformats/py-multicodec
• ipld
  • specs: ???
  • discussion: https://github.com/ipfs/go-ipld/issues/8
  • go impl: https://github.com/ipfs/go-ipld
  • py impl: https://github.com/ipld/py-ipld-dag

I know we already have several multihash implementations, and I hacked together a multiaddr implementation that could use some love. If anyone has already worked on any of these or is interested on working on any of them, please comment here. Also if anyone has anymore links to references or specs for the generic modules, post them here.

[ivilata]

I’ve uploaded pymultihash to PyPi. It’s feature complete, with some testing (using doctests) and a pretty complete tutorial in the package docstring (which I’m working to put through Sphinx). You’re very welcome to have a look at it and give your opinions, thanks! Edit: docs available in ReadTheDocs.

[ivilata]

For hashlib compatibility we only need to provide a single class, and in my opinion it still makes sense to provide some additional functionality related with multihash in the same module.

For instance (and taking much inspiration from the examples above), a single multihash.py module may be created that provides a hashlib-compatible class (hash) and a structured utility class (Multihash) in the same place:

>>> import multihash
>>> hash = multihash.hash('sha1')  # hashlib-compatible object
>>> hash.update(b'foo')
>>> digest = hash.digest()
>>> digest
b'\x11\x14\x0b\xee\xc7\xb5\xea?\x0f\xdb\xc9]\r\xd4\x7f<[\xc2u\xda\x8a3'

>>> import base58
>>> base58.b58encode(digest)  # basic way to encode a digest
'5dqx43zNtUUbPj97vJhpHyUUPyrmXG'

>>> mh = multihash.Multihash.from_digest(digest)  # parse the digest
>>> mh
multihash.Multihash(func=<multihash.Func.sha1: 17>, length=20, digest=b'\x0b...')
>>> mh.encode('base58')  # same result as above
'5dqx43zNtUUbPj97vJhpHyUUPyrmXG'

# Truncating while encoding may be supported
# but it breaks ``Mh.decode(mh.encode(X), X) == mh``,
# thus I prefer creating a different Multihash explicitly.
>>> mh.encode('base58', length=10)
'KegLCXHeZVecdh4Y'
>>> mh_trunc = mh.truncate(10)  # truncate to a new multihash
>>> mh_trunc
multihash.Multihash(func=<multihash.Func.sha1: 17>, length=10, digest=b'\x0b...')
>>> mh_trunc.encode('base58')  # same result as above
'KegLCXHeZVecdh4Y'

>>> mh_dec = multihash.Multihash.decode('KegLCXHeZVecdh4Y', 'base58')  # decoding
>>> mh_dec
multihash.Multihash(func=<multihash.Func.sha1: 17>, length=10, digest=b'\x0b...')
>>> mh_dec == mh_trunc
True
# No ``multihash.check()``, use ``from_digest()`` and catch ``ValueError``.

>>> mh.verify(b'foo')  # verification
True
>>> mh_trunc.verify(b'foo')
True

The Multihash class above could well be implemented as a namedtuple for memory savings. I used an Enum for the hash function as a comfortable and compact way to have the function code and name together, although numbers and strings may be accepted as well:

>>> multihash.Multihash(multihash.Func.sha1, 20, b'...')  # function by enum
multihash.Multihash(func=<multihash.Func.sha1: 17>, length=20, digest=b'...')
>>> multihash.Multihash(0x11, 20, b'...')  # function by code
multihash.Multihash(func=<multihash.Func.sha1: 17>, length=20, digest=b'...')
>>> multihash.Multihash('sha1', 20, b'...')  # function by name
multihash.Multihash(func=<multihash.Func.sha1: 17>, length=20, digest=b'...')
>>> multihash.Multihash(0x04, 16, b'...')  # user-defined hashing function
multihash.Multihash(func=4, length=16, digest=b'...')
>>> multihash.Multihash(0x15, 16, b'...')  # unknown hashing function
ValueError: ...

Top-level utility functions may be defined to use encoded multihash strings straight away instead of Multihash (e.g. mh_digest == b'\x11\x14\x0b...' and mh_string == '5dqx...':

multihash.encode(mh_digest, coding[, length]) -> mh_string
multihash.decode(mh_string, coding) -> mh_digest
multihash.verify(mh_string, coding, data) -> bool

But I don’t much see the point of these last functions.