Error 1114: The table 'run_details' is full"

What steps did you take:

Kubeflow was already running ~50 pipelines at the time when new pipeline runs were submitted resulting in about ~500 pipeline runs running. Volume size : 20 gigs, where mysql deployment was running.

What happened:

Pipeline runs failed to run with error :

{"error":"Failed to create a new run.: InternalServerError: Failed to store run abc-xyz to table: 
Error 1114: The table 'run_details' is full","message":"Failed to create a new run.: 
InternalServerError: Failed to store run abc-xyz to table: Error 1114: The table 'run_details' is full",
"code":13,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal Server Error",
"error_details":"Failed to create a new run.: InternalServerError: Failed to store run abc-xyz to table:
 Error 1114: The table 'run_details' is full"}]}

Kubeflow UI was also unresponsive when trying to check a pipeline run status.

What did you expect to happen:

Kubeflow UI should return/display appropriate error. Installation instructions should have this limitation documented. Also there should be a way to auto-clean this particular table or any other table which might be taking large amount of storage to keep KFP functional.

Environment:

KFP version: 1.0.1

KFP SDK version: 1.0.1

Anything else you would like to add:

checking mlpipelines.run_details table showed that table size has grown to 20 gigs.

/kind bug /area frontend /area backend

[Bobgy]

Immediate workaround could be resize the PV backing your mysql DB or delete old run records from the DB using KFP SDK/API

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment “/reopen” to reopen it.