DataFrameWriterV2 doesn't create partitions when using partitionedBy

When I try using the DataFrameWriterV2 to create an Iceberg table, the partitioning columns seem to get ignored (thereby not partitioned) even though they are specified.

Spark version: 3.1.2 Iceberg version: 0.13.0

Note: The application is written in python.

orders_df.writeTo("iceberg_catalog.p2.orders").using("iceberg").option(
    "fanout-enabled", "true"
).partitionedBy("date(pickupts)", "appname").create()

Using Spark SQL to describe the table using this way shows this:

# Partitioning
Not partitioned

The workaround for creating the tables with the appropriate partitions is via Spark SQL.

i.e. this works as expected (see note below snippet):

orders_df.createOrReplaceTempView("p2_orders_view")
spark.sql(
    """
    create table iceberg_catalog.p2.orders using iceberg
    PARTITIONED BY (date(pickupts), appname)
    as select * from p2_orders_view
    """
)

Here, when using Spark SQL to describe the table using this above method to create the table shows this:

# Partitioning
Part 0	days(pickupts)
Part 1	appname

My question is, am I using the DataFrameWriterV2 correctly when specifying the partitionedBy columns?

[liuchunhua]

@krisvaz @hililiwei

https://github.com/apache/spark/blob/v3.1.2/python/pyspark/sql/readwriter.py#L1496-L1525 missing “self._jwriter.partitionedBy(col, cols)”

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as ‘stale’