Index.to_frame() seems not working properly.

>>> kdf = ks.DataFrame({"Koalas": [1, 2, 3]}, index=pd.Index([1, 2, 3]))
>>> kdf["NEW"] = ks.Series([100, 200, 300])
>>> kdf
   0    NEW
1  1  200.0
3  3    NaN
2  2  300.0

The above code is working well.

But the same shape of DataFrame which is made from Index.to_frame() seems not work properly.

>>> kdf = ks.Index([1, 2, 3]).to_frame(name="Koalas")
>>> kdf["NEW"] = ks.Series([100, 200, 300])
>>> kdf
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: Reference 'Koalas' is ambiguous, could be: Koalas, Koalas.;

[ueshin]

InternalFrame doesn’t work only with spark_frame but with all the metadata. The data_spark_columns contains all the changes since the last time spark_frame is created.

Even if spark_frame shows the values of [1, 2, 3], data_spark_columns has the operation of kser.where(kser < 2).

>>> new_kser._internal.spark_frame.show()
+-----------------+---+-----------------+
|__index_level_0__|  0|__natural_order__|
+-----------------+---+-----------------+
|                0|  1|      25769803776|
|                1|  2|      60129542144|
|                2|  3|      94489280512|
+-----------------+---+-----------------+

>>> new_kser._internal.data_spark_columns
[Column<b'CASE WHEN CASE WHEN ((0 < 2) IS NULL) THEN false ELSE (0 < 2) END AS `0` THEN 0 ELSE NaN END AS `0` AS `0`'>]

We always use both to show the actual values.

>>> new_kser._internal.spark_frame.select(new_kser._internal.data_spark_columns).show()
+---+
|  0|
+---+
|1.0|
|NaN|
|NaN|
+---+

[itholic]

Close since this is resolved.