[FR] Show waveform for audio files in the ArtifactViewer

Willingness to contribute

Yes. I can contribute this feature independently.

Proposal Summary

For browser playback-supported audio files that are logged as artifacts, the MLflow ArtifactViewer should display a way to playback the audio file accompanied with a waveform visualized.

Motivation

What is the use case for this feature?

Researchers in things like text to speech or music information retrieval can quickly debug issues in their model’s generated files from the waveform visualization.

Why is this use case valuable to support for MLflow users in general?

Visualizing the audio artifacts’ waveforms in MLflow helps in spending less time downloading artifacts to view properties about them in other programs like digital audio workstations.

Why is this use case valuable to support for your project(s) or organization?

I am doing research in the audio domain with collaborators at school and it could save us the hassle of not having to remember where we placed all of our generated audio files and know we can always play them back in our experiments.

Why is it currently difficult to achieve this use case?

No major difficulties.

Details

• Can use the wavesurfer.js library for easy integration
• Instead of building a full media playback control component, a play/pause button and a timecode will suffice
• Can handle position the audio playhead by clicking the waveform directly

EDIT: I did not mean to cover the artifact URI and file size label in my mockup. My idea is to have the play/pause button and timecode below the label that shows the URI and file size.

What component(s) does this bug affect?

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow’s components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

What language(s) does this bug affect?

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[dbczumar]

Hi @nnethery, this sounds like a great extension for the artifact viewer. Feel free to file a PR whenever it’s convenient, and we’d be very excited to review it.

cc @xanderwebs @sunishsheth2009

[torakyun]

Hello. Hope you are doing well! This extension is great so I would like to use asap. Is there any update or a plan to release?