API for attaching user metadata to the execution plan and event

Feature

Allow pass custom key value pairs from spark job which is sent along with lineage data either in executionPlan or executionEvent. This will be powerful feature to allow users to add some metadata to the lineage. I am not sure if this feature already exists as I can see a property called extraInfo: Map[String, Any] = Map.empty in ExecutionPlan which looks like it is may be used for this purpose.

Background

The current immediate requirement is to have JobId and RunId passed as part of lineage data.

JobId: Is essentially just a unique name for the notebook that runs as job. Using Azure Databricks the applicationName and applicationId is autogenerated. These are cluster specific and not “job” specific.

RunId: is unique for a run of a job. If there are two write operation in the job then two executionPlan is generated. There is no way I can see to tell whether the two executionPlan is from same job running once (meaning there are two writes) or the job running twice (single write).

[wajda]

New API example:

spark.enableLineageTracking(new DefaultSplineConfigurer(conf) {
  override protected def userExtraMetadataProvider = new UserExtraMetaDataProvider {
    override def forExecEvent(event: ExecutionEvent, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forExecPlan(plan: ExecutionPlan, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forOperation(op: ReadOperation, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forOperation(op: WriteOperation, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forOperation(op: DataOperation, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
  }
})

There is also NoopUserExtraMetaDataProvider class with all forXXXX() methods returning empty Map. You can extend that class and only override methods that you need.

For codeless mode the following property could be used to instantiate custom UserExtraMetaDataProvider:

spline.user_extra_meta_provider.className=com.my.FooBarExtraMetaDataProvider

[ankitbko]

Just updating… got it working. Had to implement StandardSplineConfigurationStack as its not present in 0.4 and the same logic is in a private function.

Also for additional flexibility, I read one property from sparkconf which is semicolon separated keys which are then read and passed with executionPlan