When using ks.readcsv, Koalas type inference differs from pandas due to the use of Spark's inferSchema logic

Hi guys,

I’m using Koalas to combine two sets of similar data, and there is a step where we block rows using a direct comparison on a specific column. As part of this, we remove all NaN values from that column using DataFrame.dropna, but in our tests, we found that for float columns, Koalas would convert that row to an int if the values did not have decimal components. This is not exactly undesirable per se, but it IS different from the behaviour in pandas, so I thought I’d at least double check if this is intended behaviour.

Test data:

ID,name,birth_year,hourly_wage,address,zipcode
b1,Mark Levene,1987,29.5,"108 Clement St, San Francisco",94107
b2,Bill Bridge,1986,32,"3131 Webster St, San Francisco",
b3,Mike Franklin,1988,27.5,"1652 Stockton St, San Francisco",94122
b4,,1982,26,"108 South Park, San Francisco",
b5,Alfons Kemper,1984,35,"170 Post St, Apt 4,  San Francisco",94122
b6,Michael Brodie,1987,32.5,,94107

Test script:

import pandas as pd
from databricks import koalas as ks

csv_file = 'test.csv'

pandas_A = pd.read_csv(csv_file)
koalas_A = ks.read_csv(csv_file)

print('pandas_A:\n{}\n'.format(pandas_A.to_string()))
print('koalas_A:\n{}\n'.format(koalas_A.to_string()))

pandas_A_dropna = pandas_A.dropna(subset=['zipcode'])
koalas_A_dropna = koalas_A.dropna(subset=['zipcode'])

print('pandas_A_dropna:\n{}\n'.format(pandas_A_dropna.to_string()))
print('koalas_A_dropna:\n{}\n'.format(koalas_A_dropna.to_string()))

print('pandas_A_dropna[zipcode].dtype = {}'.format(pandas_A_dropna['zipcode'].dtype))
print('koalas_A_dropna[zipcode].dtype = {}'.format(koalas_A_dropna['zipcode'].dtype))

Output:

pandas_A:
   ID            name  birth_year  hourly_wage                             address  zipcode
0  b1     Mark Levene        1987         29.5       108 Clement St, San Francisco  94107.0
1  b2     Bill Bridge        1986         32.0      3131 Webster St, San Francisco      NaN
2  b3   Mike Franklin        1988         27.5     1652 Stockton St, San Francisco  94122.0
3  b4             NaN        1982         26.0       108 South Park, San Francisco      NaN
4  b5   Alfons Kemper        1984         35.0  170 Post St, Apt 4,  San Francisco  94122.0
5  b6  Michael Brodie        1987         32.5                                 NaN  94107.0

koalas_A:
   ID            name  birth_year  hourly_wage                             address  zipcode
0  b1     Mark Levene        1987         29.5       108 Clement St, San Francisco  94107.0
1  b2     Bill Bridge        1986         32.0      3131 Webster St, San Francisco      NaN
2  b3   Mike Franklin        1988         27.5     1652 Stockton St, San Francisco  94122.0
3  b4            None        1982         26.0       108 South Park, San Francisco      NaN
4  b5   Alfons Kemper        1984         35.0  170 Post St, Apt 4,  San Francisco  94122.0
5  b6  Michael Brodie        1987         32.5                                None  94107.0

pandas_A_dropna:
   ID            name  birth_year  hourly_wage                             address  zipcode
0  b1     Mark Levene        1987         29.5       108 Clement St, San Francisco  94107.0
2  b3   Mike Franklin        1988         27.5     1652 Stockton St, San Francisco  94122.0
4  b5   Alfons Kemper        1984         35.0  170 Post St, Apt 4,  San Francisco  94122.0
5  b6  Michael Brodie        1987         32.5                                 NaN  94107.0

koalas_A_dropna:
   ID            name  birth_year  hourly_wage                             address  zipcode
0  b1     Mark Levene        1987         29.5       108 Clement St, San Francisco    94107
2  b3   Mike Franklin        1988         27.5     1652 Stockton St, San Francisco    94122
4  b5   Alfons Kemper        1984         35.0  170 Post St, Apt 4,  San Francisco    94122
5  b6  Michael Brodie        1987         32.5                                None    94107

pandas_A_dropna[zipcode].dtype = float64
koalas_A_dropna[zipcode].dtype = int32

[itholic]

I think it’s time for Koalas to consider to keep the type but interprets as None or pd.NA.

totally agree. maybe related with #1203

[HyukjinKwon]

Missing value being float is known issue and that’s one of thing even pandas authors don’t quite like. So far, Koalas tried to copy this behaviour but In pandas 1.0.0, it targets to replace np.nan (https://pandas.pydata.org/pandas-docs/version/1.0.0/whatsnew/v1.0.0.html#experimental-na-scalar-to-denote-missing-values).

I think it’s time for Koalas to consider to keep the type but interprets as None or pd.NA.