Importing a new package that doesn't exist in `Pyodide`

I have been trying to look into ways to import torch but I couldn’t manage to do so because it isn’t in Pyodide’s list of supported packages.

From my research on this, there are two ways to have any new package be utilized:

1. Find a pure Python 3 wheel which means the package could be installed without the need of hard specifications.
2. Add torch and its dependencies to Pyodide’s list of support packages. This will require doing the following:
   1. Obtaining `Source Distribution of the package.
   2. Compiling the package locally.
   3. Building Pyodide from the source.
   4. Add the package to Pyodide’s list of supported packages by following the documentation
   5. Importing the newer version of Pyodide to Pyscript I don’t know how this can be done

Questions:

1. How can I add a newer version of Pyodide which was built locally to Pyscript?
2. Isn’t there a way to import a python environment that has all of the packages already installed?
3. Why do we need to have pure Python 3 wheel (aka needs to be compiled on all hardware)? Is it because it is actually being compiled on the browser-side which should be able to run all hardware?
4. From looking into the logs, it seems that every time the page has loaded the packages listed to be used get installed by micropip, is this correct? If yes, why can’t we store already installed packages to load them?
5. Why PyScript is super slow? I am assuming this because of the installation of packages from scratch (if my assumption is true).

Relevant resource:

• List of supported packages by Pyodide
• How pyenv tag is just wrapper to micropip
• PyTorch support in Pyodide issue
• GPU support for Pyodide issue
• Run pytorch in pyodide?
• Creating a Pyodide package

[verhulstm]

hi, i am excited to see your interest in this project. 😃

i have been working on getting torch into Pyodide. it’s pretty challenging.

we do have a alpha workflow to get non-pure python packages with c/c++/cython extensions into the pyscript/Pyodide ecosystem, but torch is extra complicated.

i plan on posting the build process for non-pure python packages as soon as i can

[rth]

Even more exciting from my perspective would be JAX. Are you working on this in a public repo?

There was a discussion about JAX in https://github.com/pyodide/pyodide/issues/2198, tensorflow in https://github.com/pyodide/pyodide/issues/50 and torch in https://github.com/pyodide/pyodide/issues/1625. It would be great to have them but they are all fairly large and complex packages, so it would take a lot of work to make that happen.

I would suggest you take a look at onnx

For ML wrapping JS libraries (such as ONNX Runtime Web, Tensorflow.js etc) from Python via the Pyodide’s FFI is indeed another solution. Though your API is then not going to be exactly the same as when using e.g. Tensorflow from Python, so one cannot reuse code without changes.