Next Steps

Since #1 is clogged with all the many comments I open a new issue here. Feel free to continue the discussion below and I’ll keep the following updated as things develop. Also feel free to create separate issues / repos to coordinate and I’ll add the relevant links below.

Next steps (networking, stalled – please see the “storage” section below):

• We could use some documentation for the py-libp2p library
  • See issue https://github.com/zixuanzh/py-libp2p/issues/35 for status
  • Current work: https://github.com/libp2p/py-libp2p/pull/330
• Documentation for the Bitswap protocol would be very useful as well – and not just for py-ipfs
  • See issue https://github.com/ipfs/js-ipfs-bitswap/issues/21 for status
  • Currently available information:
    • Protocol: https://github.com/ipfs/specs/blob/master/BITSWAP.md
    • API: https://ipfs.github.io/js-ipfs-bitswap/#bitswapget
• Finally, some documentation on the currently used DHT would be nice
  • See https://github.com/ipfs/camp/blob/master/DEEP_DIVES/02-scaling-up-the-dht.md for some relevant links and current developments
• Implement the Bitswap protocol in Python 3
  • Current py-ipfs-bitswap library: https://github.com/AliabbasMerchant/py-ipfs-bitswap
  • Current status/discussion: https://github.com/AliabbasMerchant/py-ipfs-bitswap/issues/1
  • → Having this would allow us to implement the ipfs block * API for fetching blocks of nodes we are connected to – fetching blocks of non-connected nodes needs the DHT.
  • To interact with go-IPFS you can start it with ipfs daemon --disable-transport-encryption, but note that you will not be able to connect to any regular peers until one of the transport encryption methods is implemented
• (Low priority) Improve the multistream-select code of py-libp2p to support actually dialing other nodes (MOSTLY FIXED UPSTREAM – ls is still missing and an mss-nc implementation could still be useful)
  • Write a mss-nc like utility on top of this code to demonstrate that you are able to connect to go-ipfs nodes and negotiate
  • Here’s some very simple sample code demonstrating the main mode of MSS:

    import socket
    s = socket.socket(socket.AF_INET)
    s.connect(("127.0.0.1", 4001))  # The connect will already exist
    s.sendall(b'\x13/multistream/1.0.0\n') # Send your supported version of MSS
    s.recv(1024) # → b'\x13/multistream/1.0.0\n' – Receive supported version of MSS by other party & validate!
    s.sendall(b"\x0d/secio/1.0.0\n") # Request the protocol you'd like to upgrade too
    s.recv(1024) # → b'\0x0d/secio/1.0.0\n…' – Confirmation that protocol is available + Protocol data OR
                 # → b'\0x03na\n' – Protocol was Not Available
    

  • The binary values at the start are varints and you need to read them byte-by-byte until you’re done decoding them, then read the remainder of each message lines based on the received length value To do this you’ll need to create an async stream based version of https://github.com/fmoo/python-varint/blob/master/varint.py
  • Additionally there is also a special ls mode in which MSS will return a list of supported protocols, see https://github.com/multiformats/multistream-select/blob/master/README.md for the complete spec
  • Subtask: Figure out how to actually dial a node using py-libp2p and document this using an example program.
• Implement SecIO libp2p transport security (easier than TLS, but will be phased out eventually)
  • You’ll need to coordinate with the py-libp2p guys on transport security modules are added exactly
  • Some background with crypto is highly recommended!
• Implement DHT peer lookups in libp2p
  • Currently available documentation: https://github.com/libp2p/specs/tree/8b89dc2521b48bf6edab7c93e8129156a7f5f02c/kad-dht
  • Go Implementation: https://github.com/libp2p/go-libp2p-kad-dht
  • JavaScript Implementation: https://github.com/libp2p/js-libp2p-kad-dht
  • For the current status see: https://github.com/libp2p/py-libp2p/issues/150 (also see the referenced PRs on that issue)
  • Interesting PRs: #129, #153, #157
  • Contacts: @alexh @zaibon @zixuanzh
• (Stretch goal) Factor out https://github.com/zixuanzh/py-libp2p/tree/master/protocol_muxer into a separate py-multistream-select library and update py-libp2p to use it (Easy!, stalled – needs your help!)
  • Current py-multistream-select library: https://github.com/dheatovwil/py-multistream-select
  • Current status: https://github.com/zixuanzh/py-libp2p/pull/101
• (Stretch goal) Implement TLSv1.3 libp2p transport security
  • You’ll need to coordinate with the py-libp2p guys on transport security modules are added exactly
  • Some background with crypto/X.509/TLS is highly recommended!

Next steps (storage, simpler):

1. Port https://github.com/ipfs/py-datastore to Python 3
   (Suggestion: Use Python’s lib2to3 and just drop Python 2 entirely.)
   • ~~Current port: https://github.com/dheatovwil/datastore~~
2. Convert datastore to use async/await using some library (maybe https://pypi.org/project/aiofiles/ ?) for file access The trio framework is used for async I/O now
3. Implement a https://github.com/ipfs/go-ds-flatfs compatible backend for the above library
4. Write a minimal py-ipfs “implementation” that can fetch blocks from the local $IPFS_PATH directory and expose them with an API similar to what https://github.com/ipfs/py-ipfs-http-client currently offers (goal here is to eventually have a drop-in replacement)
   • In progress by @alexander255 (no public code yet, most work happens in py-datastore)
5. Implement a simple Python HTTP server that emulates the block/{get,put,rm,stat} API that serves blocks from the local $IPFS_PATH directory
   • Recommendation: Use the trio-quart ASGI web microframework for this. (Whatever you choose it will have to be compatible with trio as that is the AIO framework used in the stack.)
6. (Stretch goal) Implement a badgerds compatible backend for py-datastore
   • There is an issue requesting Python bindings for the Go library, but no work has been done yet:
     dgraph-io/badger#984
7. Beyond: Start integrating IPLD to expose the UnixFS files stored in those raw blocks…

[fubuloubu]

Out of sheer morbid curiousity, has anyone thought about re-thinking this library as a light wrapper around the C++, Rust, or Go implementations instead? (Using C API, PyO3, or gopy)

Python is a great systems integration language, but a pure Python implementation seems like it’d be very slow and lagging behind the other compiled implementations with more funding.

---

Also, I am aware of the HTTP client library… just seemed like a direct integration with Python bindings might be safer with less overhead than communicating over HTTP. I’ve not had a great experience with the HTTP client either.

[MirceaKitsune]

Also, 1 more thing. What is the exact purpose of py-ipfs? We all know python is slow. So we are definitely not trying to replace the go and js versions. So, what is the exact goal?? If we know the goal, we can write code and documentation accordingly.

Python support will be very important. It’s one of the most popular scripting languages after JavaScript and used by numerous projects. To name just a few from my area of knowledge: The Blender 3D animation software and Godot game engine both use Python (named GDScript for the later) for addons and development. With a native Python implementation of IPFS, the daemon could be included in such software allowing it to directly work with files within the IPFS network… this is just one huge advantage I can immediately point out.