[BUG] Double file scan with stats skipping

Bug

Describe the problem

File stats skipping is causing two filesForScan operations.

Steps to reproduce

from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder.getOrCreate()

df = spark.range(5).select(F.struct(F.lit('test').alias('test'), F.col('id').cast("string").alias('id')).alias('nested'))
df.write.format('delta').save('test')

spark.read.format('delta').load('test').filter('nested.id = "2"').count()

Observed results

There are two scan stages, caused by hitting this line of code. Example output:

Prepared scan does not match actual filters. Reselecting files to query.
Prepared: ExpressionSet((nested#457.id = 2), isnotnull(nested#457))
Actual: ExpressionSet(isnotnull(nested#457), (nested#457.id = 2))

Expected results

The expression sets are the same so it shouldn’t trigger a second file scan.

Further details

Environment information

• Delta Lake version: 1.2.0
• Spark version: 3.2.1
• Scala version: 2.12

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

• Yes. I can contribute a fix for this bug independently.
• Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
• No. I cannot contribute a bug fix at this time.

[Kimahriman]

Ok figured some things out. There seems to be two issues when using nested columns.

1. prepared filters are pre-schema pruning, and actual filters are after. This can make the ordinal value and child schema of GetStructField different when comparing expressions
2. The actual seems to include the optional name of GetStructField, while prepared doesn’t this is removed in canonicalization, so it’s just a schema pruning issue I think

Sample code to reproduce:

from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder.getOrCreate()

df = spark.range(5).select(F.struct(F.lit('test').alias('test'), F.col('id').cast("string").alias('id')).alias('nested'))
df.write.format('delta').save('test')

spark.read.format('delta').load('test').filter('nested.id = "2"').count()

And with some extra logging where I output the contents of GetStructField:

22/04/17 08:52:09 INFO PreparedDeltaFileIndex: 
Prepared scan does not match actual filters. Reselecting files to query.
Prepared: ExpressionSet((nested#457.id = 2), isnotnull(nested#457))
Actual: ExpressionSet(isnotnull(nested#457), (nested#457.id = 2))
         
22/04/17 08:52:09 INFO PreparedDeltaFileIndex: Prepared:
22/04/17 08:52:09 INFO PreparedDeltaFileIndex: nested#457, 1, Some(id)
22/04/17 08:52:09 INFO PreparedDeltaFileIndex: Actual:
22/04/17 08:52:09 INFO PreparedDeltaFileIndex: nested#457, 0, None

[Kimahriman]

Maybe just transform GetStructField to some UnresolvedAttribute-like expression before comparing the two sets?