[BUG] MLflow fails to upload artifacts to the designated folder

Thank you for submitting an issue. Please refer to our issue policy for additional information about bug reports. For help with debugging your code, please refer to Stack Overflow.

Please fill in this bug report template to ensure a timely and thorough response.

Willingness to contribute

The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

• Yes. I can contribute a fix for this bug independently.
• Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
• No. I cannot contribute a bug fix at this time.

System information

• Have I written custom code (as opposed to using a stock example script provided in MLflow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux 5.10.105-1-MANJARO
• MLflow installed from (source or binary): source
• MLflow version (run mlflow --version): mlflow, version 1.21.0
• Python version: 3.9
• npm version, if running the dev UI: -
• Exact command to reproduce: /bin/sh -c mlflow server --backend-store-uri mysql+pymysql://${MYSQL_USER}:${MYSQL_PASSWORD}@mlflow_db:3306/${MYSQL_DATABASE} --default-artifact-root ${AWS_S3_URL} --host 0.0.0.0 (running in docker; AWS_S3_URL points to the S3 where I store the artifacts which worked in the past)

Describe the problem

I am running a MLflow (scenario 4) deployment. I have MLflow running behind a reverse proxy with http auth. My setup was working in the past. Recently, we had to generate need AWS credentials and I updated the environmental variables in my MLflow docker container and restart it (server and mysql db). Mlflow started to store the artifacts in s3://<bucket-name>/mlflow/<bucket-name>/mlflow/ instead of s3://<bucket-name>/mlflow/ which is the value of AWS_S3_URL.

When viewing a new run, the artifacts load forever and the developer console shows: Uncaught (in promise) TypeError: l.children is undefined in the ArtifactPageImpl.js:87.

I did not change the artifact path for my experiment (left it empty thus defaulting to None). This hasn’t been a problem in the past and I have a hard time to debug this scenario.

Practically, MLflow is unusable at this point!

Code to reproduce issue

Cannot post the code because my company owns it. I am basically implementing a pytorch model and want to register it in MLflow. mlflow.get_artifact_uri() prints the anticipated artifact Uri.

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

What component(s), interfaces, languages, and integrations does this bug affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow’s components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[BenWilson2]

Hi @tahesse Firstly, great job on keeping up with the debugging! 😃 Secondly, I like your idea of adding an explicit warning about reserved env variables with one caveat: let’s go one level higher and make mention in the docs of the behavior of having any unintended local ENV variables set that are reserved by MLflow for any of the backend artifact stores. Feel free to draft up a docs change (an ‘important’ tag will be good in the .rst file) and submit the PR! Thanks again!

[BenWilson2]

Thanks for clarifying @tahesse . We’ll try to get a repro going to see if we can figure out what might be going on.