metrics-collector service account doesn't exist in profile created namespace

/kind bug

Using 0.6.2 I try to launch a Katib Experiment in the namespace associated with my profile. The metrics collector jobs are not able to start because of the following error

Events:
  Type     Reason        Age                    From            Message
  ----     ------        ----                   ----            -------
  Warning  FailedCreate  2m22s (x5 over 4m53s)  job-controller  Error creating: pods "sauron-tune-191001-095415-rrfdmkpr-1569979500-" is forbidden: error looking up service account jlewi/metrics-collector: serviceaccount "metrics-collector" not found

It looks like the problem is that the metrics collector jobs are trying to use the metrics-collector service account which doesn’t automatically exist in my namespace.

There are at least two ways to fix this

1. Have profile controller create that service account
2. Use a different service account that is created by the profile controller

Work Around

To work around this issue you can manually create the service account and binding.

kubectl -n ${NAMESPACE} create serviceaccount metrics-collector

kubectl -n ${NAMESPACE} create rolebinding metrics-collector --clusterrole=metrics-collector --serviceaccount=${NAMESPACE}:metrics-collector

/cc @johnugeorge @richardsliu

[johnugeorge]

The design doc supports push based approach too(logging directly from training container) - https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1alpha3/common_types.go#L133

DB backend is made pluggable in this release(https://github.com/kubeflow/katib/pull/760) . Inorder to support KFMD(or any other DB), we just need to implement the KatibDBInterface(https://github.com/kubeflow/katib/blob/master/pkg/db/v1alpha3/common/kdb.go#L7) – Basically, Init DB, Get metric, Create metric and Delete metric functions

Currently, mysql is the only supported one. Refer: https://github.com/kubeflow/katib/blob/master/pkg/db/v1alpha3/mysql/mysql.go It should be easy to integrate any new DB backend now.

[k8s-ci-robot]

@gaocegege: Closing this issue.

In response to this:

I think so.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.