Go Templates are hard to work with programmatically

/kind feature

Describe the solution you’d like Katib currently uses go templates to define the training jobs. In the random example https://github.com/kubeflow/katib/blob/4e96b94600a598e1786ea7c88778a84747268a4d/examples/v1alpha2/random-example.yaml#L20

The training job template is set to

          apiVersion: batch/v1
          kind: Job
          metadata:
            name: {{.Trial}}
            namespace: {{.NameSpace}}
          spec:
            template:
              spec:
                containers:
                - name: {{.Trial}}
                  image: katib/mxnet-mnist-example
                  command:
                  - "python"
                  - "/mxnet/example/image-classification/train_mnist.py"
                  - "--batch-size=64"
                  {{- with .HyperParameters}}
                  {{- range .}}
                  - "{{.Name}}={{.Value}}"
                  {{- end}}
                  {{- end}}
                restartPolicy: Never

The problem with using go templates is that the manifest is no longer valid yaml. So you can’t parse the YAML spec in python and then manipulate that object programmatically.

Consider the following example

1. We are using a notebook (e.g this one) to launch katib jobs
2. We are using fairing to build the docker images
3. We would like to programmatically set the docker image in our training jobs for katib to the newly built image with fairing

Typically we would just define the K8s object a python dictionary and then we can easily set the image.

However, because the training job in the case of katib is a go template we can no longer parse it into a dictionary, manipulate it and then serialize it to YAML.

It might be worth looking at Tekton to see how it does parameter substitution since Tekton objects are valid YAML.

/cc @johnugeorge @richardsliu

[hougangliu]

yes, AST can be one another hyperparamater-inject-kind, the hyperparamater-inject-kind must keep consistent with user source code.

[gaocegege]

SGTM. Actually, I have researched if we could inject hyperparameters by rewriting Python AST.

https://github.com/caicloud/katib/blob/056cc36515052156fa2ada681d7ff487441eedc0/py/hypertransformer/README.md

NNI follows this way. I think it is better to be extensible. We could support multiples approaches.