Throwing a warning for Pandas SparseArrays

Description

Echoing OP’s sentiments from this reddit thread because it’s something I’ve had to learn the hard way as well.

Right now, sklearn secretly inflates Pandas SparseArrays without warning the user. IMO there should be a warning thrown at the very least.

Steps/Code to Reproduce

import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
df = pd.DataFrame(np.random.randn(10000, 4))
df.iloc[:9998] = 0
for col in df.columns:
    df[col] = pd.SparseArray(df[col], fill_value=0)
l = LinearRegression()
l.fit(df[df.columns[0:2]], df[df.columns[3]])

Using guppy to analyze memory usage, it’s clear that sklearn is inflating this matrix behind the scenes.

Expected Results

sklearn warns the user when inflating a sparse array

Actual Results

sklearn does not warn the user when inflating a sparse arra

Versions

System:
    python: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
executable: c:\python37\python.exe
   machine: Windows-10-10.0.18362-SP0

Python deps:
       pip: 19.2.1
setuptools: 40.8.0
   sklearn: 0.21.3
     numpy: 1.17.0
     scipy: 1.3.1
    Cython: None
    pandas: 0.25.0

[rushabh-v]

Pandas have deprecated the SparseDataframe and it suggests to create a normal Dataframe with SparseArray columns. I was thinking of determining the Sparse matrices using hasatrr for the unique attributes of the SparseDataframe. But as the inputs are Dataframe having Sparse columns there will be the same attributes as the Dataframe. Any other way to differentiate both of them?

[jorisvandenbossche]

See https://github.com/pandas-dev/pandas/issues/26706 for a discussion on how you can get for sparse columns in a dataframe, i.e. something like df.dtypes.apply(pd.api.types.is_sparse).any/all() (any/all depending on if you want to check for at least 1 or all columns being sparse)