[BUG]Spark autolog `sparkDatasourceInfo` tag not reset between runs

Willingness to contribute

The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

• Yes. I can contribute a fix for this bug independently.
• Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
• No. I cannot contribute a bug fix at this time.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Databricks 9.1LTS runtime
• MLflow installed from (source or binary): %pip install mlflow
• MLflow version (run mlflow --version): mlflow, version 1.24.0
• Python version: mlflow, version 1.24.0
• Exact command to reproduce: mlflow.spark.autolog()

Describe the problem

When I enable mlflow.spark.autolog() then trigger two different runs with with mlflow.start_run() and under each run use spark.read.load(xxx) with different data source, the run’s tag sparkDatasourceInfo will append those sources and show in the experiments. I’m expecting that different runs should clean up the tag of sparkDatasourceInfo first and then append whatever data source used within the current run.

Code to reproduce issue

mlflow.spark.autolog()
with mlflow.start_run():
    df = spark.read.format("csv").load("path1")
    display(df)
with mlflow.start_run():
    df = spark.read.format("csv").load("path2")
    display(df)

What component(s), interfaces, languages, and integrations does this bug affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow’s components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[dbczumar]

Hi @serena-ruan. Thank you for raising this! Would you be able to root cause the issue and file a fix? Happy to help provide guidance and PR review!

[serena-ruan]

Ah I see, thanks for the clarification! I’ll try to fix it this week 😄