Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Should SparseArray.astype be dense or Sparse

See original GitHub issue

Right now SparseArray.astype(numpy_dtype) is sparse:

In [6]: a = pd.SparseArray([0, 1, 0, 1])

In [7]: a.astype(np.dtype('float'))
Out[7]:
[0, 1.0, 0, 1.0]
Fill: 0
IntIndex
Indices: array([1, 3], dtype=int32)

This is potentially confusing. I did it to match the behavior of SparseSeries, but we may not want that.

Issue Analytics

State:
Created 5 years ago
Reactions:1
Comments:16 (16 by maintainers)

Top GitHub Comments

1reaction

jorisvandenbosschecommented, Oct 15, 2018

I think this is a regression compared to released version. Compare the last example:

In [18]: a = pd.SparseArray([0, 1, 0, 1])

In [20]: a.astype(bool)
Out[20]: 
[False, True, False, True]
Fill: False
IntIndex
Indices: array([1, 3], dtype=int32)

When astype-ing with a numpy dtype, I think we should simply “astype” the scalar fill_value as well.

0reactions

jorisvandenbosschecommented, Nov 8, 2018

Long term it might be nice to be able to make the distinction between astype('int') and astype(np.int64), where the first could be interpreted by pandas and keep the sparsity, while the second could ensure the output actually has dtype of np.int64. But that might also be confusing …

Top Results From Across the Web

pandas.SparseArray.astype — pandas 0.24.2 documentation

Change the dtype of a SparseArray. The output will always be a SparseArray. To convert to a dense ndarray with a certain dtype,...

Python SparseArray Dtype to Float - pandas - Stack Overflow

If you don't need sparsity anymore, use SparseArray.values.to_dense() to convert the series into a dense numpy array. The .astype() function ...

Sparse data structures — pandas 1.0.0rc0+111.ge72cd7c52 ...

A sparse array can be converted to a regular (dense) ndarray with numpy.asarray() ... From dense to sparse, use DataFrame.astype() with a SparseDtype...

Dataframe from sparse array - Dask Forum - Discourse

I want to create a sparse array with dask without ever creating a dense matrix (at least not in memory).

scipy.sparse.csr_matrix — SciPy v1.9.3 Manual

with a dense matrix or rank-2 ndarray D ... Sparse matrices can be used in arithmetic operations: they support addition, ... astype (dtype[,...