Allow dropping a column used by old SortOrders
See original GitHub issueApache Iceberg version
1.0.0 (latest release)
Query engine
Spark
Please describe the bug 🐞
Here’s my first go at a PR for this, I did it on top of a 0.9 branch. I am willing to create a proper PR if that approach seems sound.
Repro:
CREATE TABLE temp.sort_order_col_ref_drop_test (a INT, b INT);
ALTER TABLE temp.sort_order_col_ref_drop_test WRITE ORDERED BY a;
ALTER TABLE temp.sort_order_col_ref_drop_test WRITE ORDERED BY b;
ALTER TABLE temp.sort_order_col_ref_drop_test DROP column a; -- FAIL
Exception stack trace:
java.lang.NullPointerException: Cannot find source column: 1
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:953)
at org.apache.iceberg.SortOrder$Builder.addSortField(SortOrder.java:241)
at org.apache.iceberg.TableMetadata.updateSortOrderSchema(TableMetadata.java:829)
at org.apache.iceberg.TableMetadata.lambda$updateSchema$1(TableMetadata.java:460)
at org.apache.iceberg.relocated.com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:612)
at org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at org.apache.iceberg.TableMetadata.indexSortOrders(TableMetadata.java:891)
at org.apache.iceberg.TableMetadata.<init>(TableMetadata.java:282)
at org.apache.iceberg.TableMetadata.updateSchema(TableMetadata.java:464)
at org.apache.iceberg.SchemaUpdate.commit(SchemaUpdate.java:369)
at org.apache.iceberg.spark.SparkCatalog.commitChanges(SparkCatalog.java:448)
at org.apache.iceberg.spark.SparkCatalog.alterTable(SparkCatalog.java:241)
at ...
Issue Analytics
- State:
- Created 10 months ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Sort and SortByColumns functions in Power Apps
You can combine SortByColumns with a Drop down or List box control to enable users to select which column to sort by. In...
Read more >Using a sort order column in a database table - Stack Overflow
I use SQL Server 2008 R2. Edit: I need the order column of an item to be enough for sorting with no secondary...
Read more >ALTER TABLE ... DROP COLUMN allows dropping a ... - GitHub
Hm. I think that we want to allow the user to drop columns that are used by old partition specs. But that means...
Read more >Pandas Sort: Your Guide to Sorting Data in Python - Real Python
Sort a pandas DataFrame by the values of one or more columns; Use the ascending parameter to change the sort order; Sort a...
Read more >Custom Date Sort Orders in Power Pivot | Everyday Office
... called 'Sort by Column,' which allows any column to be sorted in the order of another column. In this video, we use...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Feel free to open up a PR 👍🏻
Thanks for double checking this 👍🏻