Core: Dropping an old partition column causes NPE (and corrupt metadata on v2 tables)

Apache Iceberg version

0.14.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

On a format version 2 table, dropping an old partition column on an iceberg table causes a NullPointerException in PartitionSpecBuilder, and every subsequent operation on the table throws the same exception.

java.lang.NullPointerException: Cannot find source column: 2
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:963)
	at org.apache.iceberg.PartitionSpec$Builder.add(PartitionSpec.java:517)
	at org.apache.iceberg.UnboundPartitionSpec.copyToBuilder(UnboundPartitionSpec.java:56)
	at org.apache.iceberg.UnboundPartitionSpec.bind(UnboundPartitionSpec.java:44)
	at org.apache.iceberg.PartitionSpecParser.fromJson(PartitionSpecParser.java:87)

On a v1 table, the table is still accessible after, but the ALTER TABLE throws the same NPE.

The issue is easily reproducible using the following script in a Spark Shell:

CREATE TABLE data.test_table (ts timestamp not null, day_of_ts date) USING iceberg PARTITIONED BY (day_of_ts);
ALTER TABLE data.test_table SET TBLPROPERTIES ('format-version' = '2');
ALTER TABLE data.test_table REPLACE PARTITION FIELD day_of_ts WITH days(ts);
ALTER TABLE data.test_table DROP COLUMN day_of_ts;
REFRESH TABLE data.test_table;
SELECT * FROM data.test_table;

On closer inspection of the metadata, I see that on a v1 table, the metadata is not updated when dropping the old partition field, which explains why the table is still working on v1 after, but I also don’t see what the issue is with the v2 metadata.

I am using Spark Shell on Spark 3.3.0 with Iceberg 0.14.0

[singhpk234]

As per my understanding, I think the issue here is that we are trying to bind all partition specs to to current schema which will always not be possible.

For ex : we find current schema and assign this to local var schema

https://github.com/apache/iceberg/blob/84f40cff9b98ee15b706289e551078355bb8a7a5/core/src/main/java/org/apache/iceberg/TableMetadataParser.java#L344-L351

we bind all partition specs to the schema which is equal to current schema

https://github.com/apache/iceberg/blob/84f40cff9b98ee15b706289e551078355bb8a7a5/core/src/main/java/org/apache/iceberg/TableMetadataParser.java#L381-L385

which is where I think it’s failing as we have dropped day_of_ts so current schema doesn’t have this and partition spec 0 (initial partitioning ) has reference to day_of_ts hence it’s failing to bind it.

[Fokko]

@kbendick The tests are different, it doesn’t drop the source column.

I did some digging around in the code, and we bind the partition spec right away because we need the column type for the transform:

    Builder add(int sourceId, int fieldId, String name, String transform) {
      Types.NestedField column = schema.findField(sourceId);
      Preconditions.checkNotNull(column, "Cannot find source column: %s", sourceId);
      return add(sourceId, fieldId, name, Transforms.fromString(column.type(), transform));
    }

There is a PR https://github.com/apache/iceberg/pull/5601 that will decouple this, and then we can also solve this issue.

A simple test reproduces this behavior:

  @Test
  public void testDropColumnOfOldPartitionField() {
    sql("CREATE TABLE %s (id bigint NOT NULL, ts timestamp, day_of_ts date) USING iceberg PARTITIONED BY (day_of_ts)", tableName);

    sql("ALTER TABLE %s  SET TBLPROPERTIES ('format-version' = '2');", tableName);

    sql("ALTER TABLE %s REPLACE PARTITION FIELD day_of_ts WITH days(ts)", tableName);

    sql("ALTER TABLE %s DROP COLUMN day_of_ts", tableName);
  }

I think this should be quite straightforward to fix this once https://github.com/apache/iceberg/pull/5601 has been merged