az aks upgrade --kubernetes-version leaves cluster in bad state

Related command

az aks upgrade --kubernetes-version --yes -g myrg -n myaks

Describe the bug

After upgrading from v1.22.x to v1.23.5, all subsequent operations on cluster fail.

az aks nodepool update --name nodepool1 --cluster-name myaks --resource-group myrg --enable-cluster-autoscaler --min-count 1 --max-count 3

Results in:

ERROR: (AgentPoolUpgradeVersionNotAllowed) Upgrade agent pool version from 1.23.5 to 1.22.6 is not allowed, you can only upgrade one minor version at a time. Please use [az aks get-upgrades] command to get the supported upgrade paths

az aks update -g myrg -n myaks --load-balancer-outbound-ips /resource/id

Results in:

ERROR: (NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged) Using managed cluster api, all Agent pools’ OrchestratorVersion must be all specified or all unspecified. If all specified, they must be stay unchanged or the same with control plane.

Indeed, az aks show -n myaks -g myrg -n myaks results in:

      "currentOrchestratorVersion": "1.23.5",
      "mode": "User",
      "name": "win1",
      "nodeImageVersion": "AKSWindows-2019-containerd-17763.3046.220615",
      "orchestratorVersion": "1.22.6",

However, if I use Azure Portal to rescale the nodepool in this state, this fixes the problem.

To Reproduce

1. Create cluster w/ k8s v1.22.6 with Windows nodepool.
2. Upgrade k8s version.
3. Try using ak aks update to configure outbound IP addresses.

Expected behavior

Everything works, like it would earlier. This is a new issue.

Environment summary

Running commands in Azure Cloud Shell.

{ “azure-cli”: “2.37.0”, “azure-cli-core”: “2.37.0”, “azure-cli-telemetry”: “1.0.6”, “extensions”: { “ai-examples”: “0.2.5”, “aks-preview”: “0.5.82”, “front-door”: “1.0.16”, “resource-graph”: “2.1.0”, “ssh”: “1.1.1” } }

[navba-MSFT]

@wasker I had reached out to the Product Owners in the background and they have confirmed that this is a bug. The fix will be done at the Resource Provider layer and will be released next Monday. Hope this helps.

Tagging the similar issue here https://github.com/Azure/azure-cli/issues/22807

[wasker]

Verified on my end last Thursday, no issues anymore.