Stuck on an issue?

New API example:

spark.enableLineageTracking(new DefaultSplineConfigurer(conf) {
  override protected def userExtraMetadataProvider = new UserExtraMetaDataProvider {
    override def forExecEvent(event: ExecutionEvent, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forExecPlan(plan: ExecutionPlan, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forOperation(op: ReadOperation, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forOperation(op: WriteOperation, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
    override def forOperation(op: DataOperation, ctx: HarvestingContext): Map[String, Any] = Map("foo" -> "bar")
  }
})

There is also NoopUserExtraMetaDataProvider class with all forXXXX() methods returning empty Map. You can extend that class and only override methods that you need.

For codeless mode the following property could be used to instantiate custom UserExtraMetaDataProvider:

spline.user_extra_meta_provider.className=com.my.FooBarExtraMetaDataProvider

Just updating… got it working. Had to implement StandardSplineConfigurationStack as its not present in 0.4 and the same logic is in a private function.

Also for additional flexibility, I read one property from sparkconf which is semicolon separated keys which are then read and passed with executionPlan