Azure Cosmos DB write operation support

Background

Spline is not able to track lineage while writing to Azure Cosmos DB

Feature

Azure Cosmos DB write operation support

Example

Sample Scala code of write operation to Cosmos DB, that is not able to track lineage

import com.microsoft.azure.cosmosdb.spark.schema._
import com.microsoft.azure.cosmosdb.spark.CosmosDBSpark
import com.microsoft.azure.cosmosdb.spark.config.Config
import org.apache.spark.sql.SaveMode

val customers_dataset_config_map = Map(
  "Endpoint" -> "<cosmos-db-endpoint>",
  "Masterkey" -> dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-cosmos-db-masterkey>"),
  "Database" -> "<cosmos-db-database>",
  "Collection" -> "<cosmos-db-container>",
  "WritingBatchSize" -> "100")

val customers_dataset_config = Config(customers_dataset_config_map)

// Write the output to Azure Cosmos DB
customers_dataset.write.mode(SaveMode.Overwrite).cosmosDB(customers_dataset_config)

[wajda]

0.6+

[prabdeb]

Thanks a lot for considering this feature request @wajda . Will it be available for 0.4 and 0.3 both?