PyData/Sparse support

Describe the workflow you want to enable

Replacing scipy.sparse arrays with PyData/Sparse in scikit-learn.

Describe your proposed solution

Hello. I’m the maintainer of PyData/Sparse. So far, I feel the conversation between us and the scikit-learn developers has been mostly second-hand.

I’m opening this issue in hopes of opening up a channel, and creating a meta-issue over in the PyData/Sparse repo for any requirements that scikit-learn may have to bring this to fruition. Over time, my hope is we can resolve issues on both sides to enable this.

[rgommers]

Realistically I don’t see us transitioning to pydata/sparse entirely without very extensive testing,

I agree. There’s also performance to worry about - pydata/sparse is not yet on par with scipy.sparse. And the sparse linalg support is still going through scipy.sparse.linalg anyway.

PyData/Sparse could be a soft, rather than a hard, dependency until Numba gains PyPy support.

I think there’s much more to that than PyPy support. Numba isn’t mature/stable enough to be a hard dependency yet. I don’t think much has fundamentally changed there since the discussion on making Numba a dependency of SciPy in 2018 and for Pandas in 2019. What Pandas is doing right now - accepting user-defined kernels written with Numba as input - seems fine. Numba is great for writing individual functions that are fast easily. But for a hard dependency, you’d expect things like non-super-scary-exceptions, portability, stability, etc.

[amueller]

Ah ok, that will probably work then, though I assume we’ll have to write some conversion tools.