[BUG] OSError: [Errno 30] Read-only file system

Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

MLflow version

1.27.0

System information

NAME=“Ubuntu” VERSION=“20.04.4 LTS (Focal Fossa)” as underlying OS

Running Docker version 20.10.12, build 20.10.12-0ubuntu2~20.04.1 docker-compose version 1.25.0

FROM python:3-slim
ARG MLFLOW_VERSION=1.27.0

WORKDIR /mlflow/

COPY ./requirements.txt ./requirements.txt

RUN pip install -r requirements.txt

EXPOSE 5000

CMD mlflow server --backend-store-uri ${BACKEND_URI} --default-artifact-root ${ARTIFACT_ROOT} \ 
    --host 0.0.0.0 --port 5000 

version: '3'

services:
  mlflow:
    image: 'atcommons/mlflow-server'
    build: .
    ports:
      - "8080:5000"
    volumes:
      - ./data:/mlflow
      - ./data/mlartifacts:/mlflow/mlartifacts:rw
    environment:
      BACKEND_URI: sqlite:////mlflow/mlflow.db
      ARTIFACT_ROOT: /mlflow/mlartifacts

requirements.txt:

mlflow==1.27.0 psycopg2-binary boto3

Describe the problem

If I try to log an experiment and add artifacts using the following:

import uuid
import mlflow
model = onnx.load(f'{path}/model.onnx')
mlflow.set_experiment(uuid.uuid4().hex)

with mlflow.start_run(run_name=name):
    mlflow.get_artifact_uri()
    mlflow.log_artifact(f'{path}/MLmodel', artifact_path=name)
    mlflow.log_artifact(f'{path}/config.yml', artifact_path=name)
    mlflow.log_artifact(f'{path}/conda.yaml', artifact_path=name)
    mlflow.log_artifact(f'{path}/model_config.yaml', artifact_path=name)
    mlflow.log_artifact(f'{path}/processing.py', artifact_path=name)
    mlflow.onnx.log_model(model, artifact_path=name)

I get the error: OSError: [Errno 30] Read-only file system: '/mlflow'

If I ssh into the container I can go to the mounted file that is at /mlflow/mlartifacts and create a file manually, exit the container and see the file on the local system all without issue.

Tracking information

No response

Code to reproduce issue

see Describe the problem above

Other info / logs

No response

What component(s) does this bug affect?

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow’s components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

What language(s) does this bug affect?

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[harupy]

Looks like it’s truncated. Do you have the ... part?

[harupy]

@zstern If you want to log artifacts in the tracking server, you need to run mlflow server with --serve-artifacts and --artifacts-destination. This example might help. Feel free to reopen the issue if you need further help.