Throwing runtime exception for delta commands

Current spline is not supported for delta commands such as MergeInto, Update, Delete

I have added io.delta package with scope provided as databricks cluster by default has io.delta imported, and made changes to WriteCommandExtractor.scala file.

import org.apache.spark.sql.delta.commands._

and added below code to commandsToBeImplemented just to see if it throws custom exception when run in Required mode.

classOf[MergeIntoCommand]

But it throws following exception when I run an application

Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.commands.MergeIntoCommand

I am trying to debug the issue but your support on this is highly appreciated.

[wajda]

That’s why Spline agent is full of dirty reflection hacks 😃 We’re living in wild water…

[cerveada]

How to log is described here https://github.com/AbsaOSS/spline-spark-agent/discussions/394 especially the last paragraph. You need a TRACE level for evrything under za.co.absa.spline package.

You should find the MergeIntoCommandEdge command in an OBJECT DUMP BEGIN log. This is a log of the class structure and fields, that should hopefully be enough to see the fields you need to extract the dataSource data without seeing the actual code.

For the implementation, check the PR for Databricks’s CreateDeltaTableCommand https://github.com/AbsaOSS/spline-spark-agent/pull/112/files

It might be enough to just add some code in DatabricksPlugin similar way it is done for CreateDeltaTableCommand.