Perf. improvement in 'write()': Numpy slices are memoryless while pandas ones are not.

Hi, Just realized that this evening:

import sys
import pandas as pd
import numpy as np

# Numpy
#ar = np.arange(200_000_000)
ar = np.arange(1_000_000)

sys.getsizeof(ar)
#Out[5]: 8000104
sys.getsizeof(ar[:])
#Out[6]: 104

# Pandas
df = pd.DataFrame({'a':[ar]})

sys.getsizeof(df)
#Out[8]: 8000256
sys.getsizeof(df[:])
#Out[9]: 8000256

We are doing such slices when we are splitting new data in row groups for writing in write(). After reading some SO answers, it would seems that numpy ‘simple’ slices are direct pointers to area in the original table. I could not find information about pandas case, but it seems a ‘deep copy’ is made, meaning, it creates a new DataFrame. This takes some more memory, and some time for filling new memory slots I guess (well, the time to make a copy).

A perf. improvement would be to switch current pandas slices to numpy slices.

I will not manage this in current PR write_row_groups() (its scope is already big enough I think), but can have a look in a next PR. Bests,

[martindurant]

Pandas definitely does some extra checking and extra work (making index instances and block managers) compared to numpy, but if the memory is not copied, then it shouldn’t be significant. In many cases, perhaps the most common, a single dataframe becomes a single row-group.

[yohplala]

I am aligned with this. Closing the ticket.