[SDK] Create API to get Trial metrics from Katib DB

/kind feature /area sdk

Our Katib Python SDK doesn’t have an API to get Trial metrics from Katib DB. Currently, user can see the Trial metrics only using Katib UI. We should give an ability to query metrics using GetObservationLog gRPC API via Katib SDK.

From the security perspective user can run this gRPC API from any namespace and any experiment since our DB Manager doesn’t have any auth checks, right ? Should we investigate how to improve user isolation for Katib (“multi-user mode feature”) ? One solution could be to use Istio to allow traffic only from the appropriate user, as @apo-ger mentioned here: https://github.com/kubeflow/katib/pull/1983#issuecomment-1319674570.

What do you think @johnugeorge @gaocegege @tenzen-y @anencore94 @kimwnasptd @apo-ger ?

---

Love this feature? Give it a 👍 We prioritize the features with the most 👍

[kimwnasptd]

@andreyvelich that’s a great feature!

Regarding the authnz part, I think this discussion will revolve around having programmatic client support for the DB Manager API Server. This is the same with how KFP allows Pods from other namespaces to use its API Server to perform CRUD tasks https://github.com/kubeflow/pipelines/issues/5138.

And this is done by:

1. Allowing everyone to talk to the DB Manager, but without setting the kubeflow-userid header (to avoid impersonations).
   • The istio-ingressgateway should still be able to set this header though, since we trust that it’s setting the user header correctly
   • This is the AuthorizationPolicy in KFP for this https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/base/installs/multi-user/istio-authorization-config.yaml#L36-L38
2. The DB Manager will drop any requests that are not authenticated
3. In-cluster pods that will need to talk to the DB Manager will need to provide an audience scoped ServiceAccount token
4. The DB Manager will need to validate the token
5. The DB Manager will then extract the identity (ServiceAccount name) from that token and perform a SubjectAccessReview

Then there’s also the discussion on how to use the ServiceAccount tokens from outside the cluster. But this is a next step once we have the above in-cluster behavior working

[anencore94]

https://docs.google.com/document/d/1TRUKUY1zCCMdgF-nJ7QtzRwifsoQop0V8UnRo-GWlpI/edit?disco=AAAAknO9PlM

For answering the above question, @andreyvelich . I’ve seen many company make their own UI page using several kubeflow APIs including kubeflow notebooks, pipelines and katib. Thus if there is a http server for katib, many clients including there own sdk and ui will use those APIs much easier