pivot_table function with renaming columns

Here is the example from the documentation.

df = ks.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
                          "bar", "bar", "bar", "bar"],
                    "C": ["small", "large", "large", "small",
                          "small", "large", "small", "small",
                          "large"],
                    "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]},
                   columns=['A',  'C',  'E'])
df.pivot_table(values='E', index=['A'],
                        columns='C', aggfunc='sum')

The above code works as expected. Then I rename the columns like

df.columns =['New_A','New_C','New_E']
print(df.columns)

This works fine with applymap and other functions like to_datetime but as soon as I use pivot_table , I got an error.

df.pivot_table(values='New_E', index=['New_A'],
                        columns='New_C', aggfunc='sum')

The above code will give the following error:

KeyError: 'No StructField named New_E'

Is it not allowed to change the column name right now?? Thanks.

[harupy]

I’ll work on this.

[ueshin]

Shall we fix the columns setter to rename the Spark columns for now? @itholic @harupy Do you want to fix it?

I’ll handle merge and pivot_table soon.