CreateKnativeService does not sync to desired state in non-happy path cases

@yuzisun The current implementation of CreateKnativeService will not sync cluster state to spec desired state in the following cases (admittedly off the happy path):

1. User modifies the default spec while a canary is active. Default revision will remain unchanged
2. New cluster (does not have any existing revisions) syncs serving CRD with default and canary. Will only provision canary?

I think these issues stem from the fact that this logic depends on cluster state (revision state). Can we do some other knative thing to resolve this? Happy to try to fix if can point me in the right kn direction!

https://github.com/kubeflow/kfserving/blob/master/pkg/reconciler/ksvc/resources/knative_service.go

	if kfsvc.Spec.Canary == nil || kfsvc.Spec.Canary.TrafficPercent == 0 {
		//TODO(@yuzisun) should we add model name to the spec, can be different than service name?
		container = CreateModelServingContainer(kfsvc.Name, &kfsvc.Spec.Default)
		revisions = []string{knservingv1alpha1.ReleaseLatestRevisionKeyword}
		routingPercent = 0
	} else {
		container = CreateModelServingContainer(kfsvc.Name, &kfsvc.Spec.Canary.ModelSpec)
		revisions = []string{kfsvc.Status.Default.Name, knservingv1alpha1.ReleaseLatestRevisionKeyword}
		routingPercent = kfsvc.Spec.Canary.TrafficPercent
	}

[yuzisun]

@ellis-bigelow @rakelkar @cliveseldon, After discussing with Knative team and Ellis is right just that revisions are immutable and we need to use configuration, I am pretty sold that we can go for two configurations and one route that splits the traffic, trying out that now.

[rakelkar]

closed by #51