`mlmd.errors.InternalError` raised when duplicate entries found in metadata store.

System information

• Have I specified the code to reproduce the issue (Yes, No): No
• Environment in which the code is executed (e.g., Local(Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc): AWS with Amazon RDS as the metadata store
• TensorFlow version: 2.5.2
• TFX Version: 1.2.0
• Python version: 3.8
• Python dependencies (from pip freeze output):

...
ml-metadata==1.2.0
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.5.2
tensorflow-data-validation==1.2.0
tensorflow-estimator==2.5.0
tensorflow-hub==0.12.0
tensorflow-metadata==1.2.0
tensorflow-model-analysis==0.33.0
tensorflow-probability==0.13.0
tensorflow-recommenders==0.5.2
tensorflow-serving-api==2.5.3
tensorflow-transform==1.2.0
tfx==1.2.0
tfx-bsl==1.2.0
...

Describe the current behavior

This issue happens when a Transform component tires to insert an entry to the ML Metadata Store (Amazon RDS), if the entry exists, a mlmd.errors.InternalError is raised with errno: 1062, and the error failed to be caught by the code below, which leads to a failure for the Transform component. https://github.com/tensorflow/tfx/blob/d8b226c4c13deec668f60825e3b2d14efdf7ef77/tfx/orchestration/metadata.py#L696

Error:

mlmd.errors.InternalError: mysql_query failed: errno: 1062, error: Duplicate entry '136887-tfx_version-1' for key 'PRIMARY'"

Describe the expected behavior

When Transform component tires to insert an entry to the ML Metadata Store (Amazon RDS), if the entry exists, a mlmd.errors.AlreadyExistsError error should be raised and caught handled by TFX.

Standalone code to reproduce the issue

Name of your Organization (Optional)

Other info / logs

Our current workaround is using the following patch:

701c701,703
<     except mlmd.errors.AlreadyExistsError:
---
>     except (mlmd.errors.AlreadyExistsError, mlmd.errors.InternalError) as e:
>       if isinstance(e, mlmd.errors.InternalError) and "errno: 1062" not in e.message:
>         raise e

[parham12]

Hi there, With Apache Airflow as the orchestrator, I have the same problem when using [MLMD] 1.8.0. A log file related to this issue is attached. dag_id=consumer_complaint_pipeline_airflow_run_id=manual__2022-07-12T10_25_40.659461+00_00_task_id=Transform_attempt=1.log

[0xleowang]

Unfortunately, we only encountered this error a few times, didn’t find a way to reproduce it.