Converting NaN objects falsely turns NaN into None

When converting a DataFrame/Series of type object (i.e. Strings) with np.nan values to Koalas DataFrames and back, the former np.nan values are replaced with None as can be seen below:

>>> ks.Series(['a', np.nan]).to_pandas()
0       a
1    None
Name: 0, dtype: object

However, the following output would be expected instead:

0      a
1    NaN
Name: 0, dtype: object

I assume this behavior is caused by the fact that Spark does not support NaN values for String columns but uses None instead. Consequently, there is probably no definite way to decide whether a None value in Spark should be converted to a Python NaN or None at the time when the conversion from Spark to pandas happens. However, I would argue that in doubt converting to NaN makes more sense in most cases than None and should thus be the default.

What is your opinion on this?

[thunterdb]

Note that performance is not going to be an issue here in any case: the internal representation in spark of None is a bit mask and the scalar values are always allocated.

Here is what I suggest to do when we convert the data between pandas and spark:

• for scalar values that have an intrinsic representation of NaN (float, double): pandas -> spark: NaN is converted to non-null NaN spark -> pandas: null or non-null NaN is converted to NaN
• for scalar values that do not have an intrinsic representation of NaN (int, short, …): pandas does not allow that to begin with so there is a loss of precision here. it will depend on the behaviour of the python pickler and of arrow
• for objects (strings, structs, …): Spark can optimize the memory layout when the value is null, so: pandas -> spark: NaN -> null, None -> non-null None (I did not try, may not always work) spark -> pandas: null -> NaN non-null None -> None

@floscha does that cover all the use cases that you are thinking of? Can you write a couple of test cases to validate that this is the expected behaviour.

Also, be aware that the nullability support in spark can be brittle, some corner cases especially after UDFs or joins will forget nullability info.

[itholic]

Let me close this for now, since It can’t be supported because Spark can’t tell if the null value of object type should be None or NaN when converting to pandas anyway.