Install with pyodide

I was playing around with jupyterlite and tried to install Dask

import micropip
await micropip.install("dask")

I think that it would be fun if we could get dask and distributed running smoothly. My guess is that this is doable if we’re able to package things up in a pure python way, but that currently we’re not in some way.

cc @jakirkham who I suspect understands this space well

Traceback

Traceback (most recent call last):
  File "<console>", line 2, in <module>
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 158, in install
    transaction = await self.gather_requirements(requirements, ctx)
  File "/lib/python3.8/site-packages/micropip.py", line 154, in gather_requirements
    await gather(*requirement_promises)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 280, in __step
    result = coro.send(None)
  File "/lib/python3.8/site-packages/micropip.py", line 227, in add_requirement
    await self.add_requirement(recurs_req, ctx, transaction)
  File "/lib/python3.8/site-packages/micropip.py", line 222, in add_requirement
    wheel, ver = self.find_wheel(metadata, req)
  File "/lib/python3.8/site-packages/micropip.py", line 238, in find_wheel
    release = releases[str(ver)]
KeyError: '3.9'
PythonError: Traceback (most recent call last):
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/pyolite/interpreter.py", line 23, in run
    result = await self.run_complete
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/pyodide/console.py", line 253, in load_packages_and_run
    raise e
  File "/lib/python3.8/site-packages/pyodide/console.py", line 247, in load_packages_and_run
    result = await eval_code_async(
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 419, in eval_code_async
    return await CodeRunner(
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 278, in run_async
    res = await res
  File "<console>", line 2, in <module>
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/micropip.py", line 158, in install
    transaction = await self.gather_requirements(requirements, ctx)
  File "/lib/python3.8/site-packages/micropip.py", line 154, in gather_requirements
    await gather(*requirement_promises)
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 280, in __step
    result = coro.send(None)
  File "/lib/python3.8/site-packages/micropip.py", line 227, in add_requirement
    await self.add_requirement(recurs_req, ctx, transaction)
  File "/lib/python3.8/site-packages/micropip.py", line 222, in add_requirement
    wheel, ver = self.find_wheel(metadata, req)
  File "/lib/python3.8/site-packages/micropip.py", line 238, in find_wheel
    release = releases[str(ver)]
KeyError: '3.9'

    at new_error (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.js:9:175413)
    at wrap_exception (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[297]:0x1a6c94)
    at FutureDoneCallback_call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[379]:0x1aaab3)
    at byn$fpcast-emu$FutureDoneCallback_call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[14420]:0x7a257c)
    at _PyObject_MakeTpCall (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[760]:0x1cabed)
    at __static_1198 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[2906]:0x2c54f1)
    at byn$fpcast-emu$__static_1198 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[22640]:0x7bf866)
    at __static_678 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[1621]:0x21220e)
    at byn$fpcast-emu$__static_678 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[22120]:0x7bdce6)
    at PyVectorcall_Call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[763]:0x1cb006)

[bollwyvl]

Huzzah, looks like a bunch of the examples Just Work ™️ in jupyterlite today:

import piplite
await piplite.install('dask[array,dataframe,diagnostics]')
import dask.array as da

x = da.random.random((1000, 1000), chunks=(100, 100))
x.compute()

And the demo site even has ipycytoscape:

import dask
df = dask.datasets.timeseries()
df.visualize()

Seems like some hot examples in the docs are more than possible now, akin to what numpy, pandas, sympy and ipywidgets are doing.

[rth]

If all the dependencies are pure Python and have wheels they would indeed be uninstallable with micropip.

It looks like maybe pyyaml is the blocker here.

Indeed, it should be resolved with https://github.com/pyodide/pyodide/pull/1181

A complication is that

• tornado has some C extensions and so would need to be compiled via the Pyodide build system by adding it as a package here
• and more importantly sockets aren’t supported in the Webassembly VM : that might be a blocker for tornado?

If tornado doesn’t work (I don’t know much about how it works), I imagine, then one would need to find some alternative for communicating between nodes, possibly with WebSockets?

In the WASM VM, multiprocessing is indeed not supported, threading is not supported yet on all browsers (and not enabled in Pyodide), however there is a possibility to use WebWorkers which also allow to create separate threads although with a different API.

As a more theoretical question — what would the point be of using dask in pyodide right now?

One could be to offload expensive calculations to a cluster. Though in that respect one could also just use jupyter notebook directly with a locally running dask.

Another use case would be to be able to send commands to Python in WASM from Python running locally (or the other way around), for instance for testing https://github.com/pytest-dev/pytest-xdist/issues/336