Writing a DataFrame to an Excel file doesn't produce Operation in Spark Logical Plan

Hello, I’m trying to add support for spark-excel in our Data Lineage Tracking tool Spline.

We use Spark QueryExecutionListener to capture the logical plan, and then we create lineage from it. Unfortunately there is no operation in the plan for writing in the Excel. On the other hand the Reading from Excel is contained in the plan.

We will be adding support for reading operation, but if you manage to make spark include the write operation in plan we can add support for that in Spline as well.

// This is the code I use when trying to produce the plan:

  val df = spark.read
    .format("com.crealytics.spark.excel")
    .option("useHeader", "true") // Required
    .load("/Users/abac720/test.xlsx")

  val res = df.select($"number" + 1, $"text")

  res.write
    .format("com.crealytics.spark.excel")
    .option("dataAddress", "'My Sheet'!B3:C35")
    .option("useHeader", "true")
    .save("data/output/batchWithDependencies/result.xlsx")

[cerveada]

At the end I was able to extract lineage data for both Read and Write.

I don’t know why it didn’t work from the start, but Ii does now. So I’m closing this ticket.

[cerveada]

Ok, I will add support for the reading only now and after the new version is released we can add support for it as well.