Long term solution for AutoML CI/CD test infrastructure

/kind feature

We need to figure out a long term plan to run CI test infrastructure under AutoML projects. Right now, we use Travis to run our unit and integration tests, kubeflow-ci and automl-ci project to run our e2e tests, using Prow. During Prow test we follow these steps:

• We submit Argo workflow from ksonnet app to kubeflow-testing cluster in kubeflow-ci project. Do we want to refactor this workflow using Tekton, for example?

• For pre-submit we use gcr.io/automl-ci registry and automl-ci project to build and push images. For post-submit we use gcr.io/kubeflow-images-public registry and kubeflow-ci project to build and push images. As mentioned here: https://github.com/kubeflow/katib/pull/1326#issuecomment-689512234 we can switch to apps-cd to release our images instead of using post-submit. @gaocegege @johnugeorge Do we want to use our own registry under automl-ci project or continue use gcr.io/kubeflow-images-public ?

• For e2e test we create Kubernetes cluster under automl-ci project, deploy manifests and submit required Experiments.

Since kubeflow-ci is maintaining only for 20% (https://github.com/kubeflow/testing/issues/737) we need to figure out the way to deploy our test infra. Some possible solutions can be:

• Use Kind with GitHub actions to build images and run our e2e tests.
• Switch to AWS common test infra on which @Jeffwan is currently working.
• Continue maintain automl-ci project, create cluster there to trigger Prow jobs.

/cc @gaocegege @johnugeorge @jlewi

[jlewi]

@andreyvelich it looks like the cloudbuild service account [e-mail] for project kubeflow-ci should also have had its permissions removed as part of the lockdown of kubeflow-images-public. I have updated it to expire on 11/02

[andreyvelich]

@jlewi October 17 as a deadline looks good for us. We verified that GitHub actions can work.

If we don’t switch to AWS test infra until the deadline, we will use GitHub actions. /cc @johnugeorge @gaocegege