Failure evaluating expressions for NOT + STARTS_WITH predicate

scala> val df = spark.read.format("iceberg").load("/tmp/iceberg/logs")
df: org.apache.spark.sql.DataFrame = [level: string]

scala> df.filter(not($"level".startsWith("b"))).show()
java.lang.IllegalArgumentException: No negation for operation: STARTS_WITH
  at org.apache.iceberg.expressions.Expression$Operation.negate(Expression.java:72)
  at org.apache.iceberg.expressions.UnboundPredicate.negate(UnboundPredicate.java:69)
  at org.apache.iceberg.expressions.RewriteNot.not(RewriteNot.java:44)
  at org.apache.iceberg.expressions.RewriteNot.not(RewriteNot.java:22)
  at org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:293)
  at org.apache.iceberg.expressions.Projections$BaseProjectionEvaluator.project(Projections.java:153

There are multiple places in Iceberg which call RewriteNot on the user provided filters (1, 2, 3 and many more). However, the STARTS_WITH predicate does not support negation and hence RewriteNot throws an exception. Should we implement a NOT_STARTS_WITH operation to support this usecase?

Another approach would be to just keep NOT expression in RewriteNot for predicates that do not support negation. However some evaluators in Iceberg make the assumption that there are no NOTs in the tree and it may affect their correctness. https://github.com/apache/iceberg/blob/425a45f8acec0496d77e070c07fb209de92ab2c1/api/src/main/java/org/apache/iceberg/expressions/Projections.java#L148

cc: @rdblue @aokolnychyi

[rdblue]

I’m not. Go for it!

[cccs-eric]

I ran into the same issue while trying to run a SQL query using Spark against an Iceberg table. It crashed with the same error: IllegalArgumentException: No negation for operation: STARTS_WITH. I ran the following query in a Jupyter Notebook:

%%sparksql --view bug --cache
SELECT
    *
FROM
    namespace.my_table AS tbl
WHERE
    tbl.aDate > CURRENT_DATE() - INTERVAL 10 DAYS
    AND tbl.activityDisplayName = 'Update user'
    AND rawJSON LIKE '%SourceAnchor%'
    AND displayName NOT LIKE 'Sync%'

If I create a temporary table in Spark using one or some of the table data files and issue the same query, it works. So it really is when Iceberg is in play.