ENH: speeding up pairplot and alike

I think there might be room for speeding up methods like sns.pairplot.

I want to be able to draw huge plots (100x100 or more) and this is of course unrealistic with the current code. I wonder if we could parallelise some of the loops like this one:

https://github.com/mwaskom/seaborn/blob/445a54aa46e279406949e0a3c2eed88d6cf80223/seaborn/axisgrid.py#L649

On matplotlib’s side I am confident there would be no blocker as they have this tutorial https://matplotlib.org/stable/gallery/misc/multiprocess_sgskip.html

I am not entirely sure if this would be the correct place to do it. Let me know if this is of interest/feasible and I would be happy to help.

[mwaskom]

Here’s an extraordinarily simple proof of principle that I threw together just for exploration:

import numpy as np
import matplotlib.pyplot as plt
from joblib.parallel import Parallel, delayed

def pairplot(ax, x, y):
    ax.scatter(x, y)
    
xs = np.random.randn(4, 10000)
ys = np.random.randn(4, 10000)

f, axs = plt.subplots(1, 4, sharex=True, sharey=True)
Parallel(n_jobs=4)(delayed(pairplot)(ax, x, y) for ax, x, y in zip(axs, xs, ys))

With n_jobs=1 this works as expected but with n_jobs=4 I get four blank axes. I’ve done no further debugging or tuning.

I think I’ll close for now, not because I’m opposed to the idea of parallelizing the axisgrid objects in principle, but because I’m skeptical that it’s possible for reasons outside of seaborn’s control. (Unlike some projects that leave open any issue touching on a feature that doesn’t technically exist, I prefer to keep the issue queue relatively high signal and close issues below a certain priority threshold even if they are good ideas; I’ve revisited lots of closed issues in the past).

I’d rather consider the usefulness/feasibility of this idea in the context of an existing prototype of how one might parallelize operations over the axes in a single matplotlib figure in general.

[tupui]

Thanks for the link to the heat-scatter. I will play with that and some measures.

In the meantime, feel free to close the issue if you think this would be more something to address on matplotlib’s side. Or that this is not realistic/out of scope here.