Unknown error when runnnig experiments

/kind bug

Got unexpected error in metric collector using file collector

F1005 06:27:13.600768 14 main.go:396] Failed to Report logs: rpc error: code = Unknown desc = Pepare SQL statement failed: Error 1390: Prepared statement contains too many placeholders

The job finished properly, but showed as error in the pod status. Found the error above in the file metric collector log. How shall I debug this?

The output looks like:

objective: 0.716
loss_avg: 0.716
acc_avg: 0.500

objective: 0.704
loss_avg: 0.704
acc_avg: 0.540

Epoch eval:
ROCAUC: 0.605

Validation:
objective: 0.671
loss_avg: 0.671
acc_avg: 0.605

Epoch eval:
ROCAUC: 0.609
Validation rocauc: 0.609

time(s): 67894.691

What did you expect to happen: Works well

Anything else you would like to add: There’s warning above kind of related to this

W1005 06:27:08.292701 14 file-metricscollector.go:71] Metrics will not have timestamp since error parsing time objective:: parsing time “objective:” as “2006-01-02T15:04:05.999999999Z07:00”: cannot parse “objective:” as “2006” W1005 06:27:08.292708 14 file-metricscollector.go:71] Metrics will not have timestamp since error parsing time Validation: parsing time “Validation” as “2006-01-02T15:04:05.999999999Z07:00”: cannot parse “Validation” as “2006”

Environment:

• Kubeflow version (kfctl version): Didn’t install kf
• Minikube version (minikube version):
• Kubernetes version: (use kubectl version): 1.20
• OS (e.g. from /etc/os-release): Amazon Linux2

[VoVAllen]

Hi, I cannot share the full yaml. Below is the metricCollector and objectivepart

  metricsCollectorSpec:
    collector:
      kind: File
    source:
      filter:
        metricsFormat:
          - ([\w|-]+)\s*:\s*((-?\d+)(\.\d+)?)
      fileSystemPath:
        path: /tmp/output.log
        kind: File

and

  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: objective
    additionalMetricNames:
      - rocauc

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment “/reopen” to reopen it.