[BUG]: Updating a VMSS with new capacity leads to outdated VM models

Describe the bug When updating the scaling capacity of a Virtual Machine Scale Set via Azure Java SDK such as

vmss.update()
           .withCapacity(newCapacity)
           .apply();

the other existing VMs of the VMSS are then somehow outdated and the UI shows they do not have the latest VMSS model applied.

When however changing the scaling capacity via the UI the VMs of the VMSS still have the latest VMSS model applied.

Exception or Stack Trace There are no errors, it simply seems to be a mismatch in behavior between the API and the SDK

To Reproduce Steps to reproduce the behavior:

Code Snippet see above

Expected behavior I would suggest that simply changing the scaling count should not affect the VMSS model of the VMs itself

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: [Windows]
• IDE: [IntelliJ]
• Library/Libraries: [e.g. com.azure:azure-resourcemanager:2.20.0]
• Java version: [LibericaJDK-16]
• Frameworks: [Quarkus]

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [x ] Bug Description Added
• [x ] Repro Steps Added
• [x ] Setup information Added

[XiaofeiCao]

Thanks @hendrik-schaffer for your detailed explanation of your situation. I’ll check if the patch pattern can resolve the altering of the latest model state.

@weidongxu-microsoft This may be an issue we want to fix. When the vmss’s UpgradePolicy is set to Automatic, all existing VMs in the vmss is likely to be rebooted if the Latest Model is No. I’ll have that checked too.

[hendrik-schaffer]

Sure. Please find below the JSON View of our VMSS

{
    "name": "enterpriserunner-vmss-ubuntu-server-int",
    "id": "/subscriptions/SUBSCRIPTION-ID/resourceGroups/RG_REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/enterpriserunner-vmss-ubuntu-server-int",
    "type": "Microsoft.Compute/virtualMachineScaleSets",
    "location": "westeurope",
    "tags": {
        "createdBy": "Jenkins"
    },
    "identity": {
        "type": "UserAssigned",
        "userAssignedIdentities": {
            "/subscriptions/SUBSCRIPTION-ID/resourceGroups/RG_REDACTED/providers/Microsoft.ManagedIdentity/userAssignedIdentities/githubrunnervms-mi-int": {
                "principalId": "REDACTED",
                "clientId": "REDACTED"
            }
        }
    },
    "sku": {
        "name": "Standard_DS2_v2",
        "tier": "Standard",
        "capacity": 3
    },
    "properties": {
        "singlePlacementGroup": true,
        "orchestrationMode": "Uniform",
        "upgradePolicy": {
            "mode": "Manual",
            "rollingUpgradePolicy": {
                "maxBatchInstancePercent": 20,
                "maxUnhealthyInstancePercent": 20,
                "maxUnhealthyUpgradedInstancePercent": 20,
                "pauseTimeBetweenBatches": "PT0S"
            }
        },
        "virtualMachineProfile": {
            "osProfile": {
                "computerNamePrefix": "atcrun",
                "adminUsername": "cloud",
                "linuxConfiguration": {
                    "disablePasswordAuthentication": true,
                    "ssh": {
                        "publicKeys": [
                            {
                                "path": "/home/cloud/.ssh/authorized_keys",
                                "keyData": "ssh-rsa SSH_KEY_REDACCTED"
                            }
                        ]
                    },
                    "provisionVMAgent": true
                },
                "secrets": [],
                "allowExtensionOperations": true,
                "requireGuestProvisionSignal": true
            },
            "storageProfile": {
                "osDisk": {
                    "osType": "Linux",
                    "diffDiskSettings": {
                        "option": "Local",
                        "placement": "CacheDisk"
                    },
                    "createOption": "FromImage",
                    "caching": "ReadOnly",
                    "managedDisk": {
                        "storageAccountType": "Standard_LRS"
                    },
                    "diskSizeGB": 30
                },
                "imageReference": {
                    "id": "/subscriptions/SUBSCRIPTION-ID/resourceGroups/RG_REDACTED/providers/Microsoft.Compute/galleries/atc_images_int/images/enterpriserunner-ubuntu-20.04-gen2-int/versions/3.6.1"
                }
            },
            "networkProfile": {
                "networkInterfaceConfigurations": [
                    {
                        "name": "entercfecNic",
                        "properties": {
                            "primary": true,
                            "enableAcceleratedNetworking": false,
                            "disableTcpStateTracking": false,
                            "dnsSettings": {
                                "dnsServers": []
                            },
                            "enableIPForwarding": false,
                            "ipConfigurations": [
                                {
                                    "name": "entercfecIPConfig",
                                    "properties": {
                                        "subnet": {
                                            "id": "/subscriptions/SUBSCRIPTION-ID/resourceGroups/RG_REDACTED/providers/Microsoft.Network/virtualNetworks/VNET-SPOKE-AMS-0221/subnets/Subnet-1"
                                        },
                                        "privateIPAddressVersion": "IPv4"
                                    }
                                }
                            ]
                        }
                    }
                ]
            },
            "extensionProfile": {
                "extensions": [
                    {
                        "name": "CustomScript",
                        "properties": {
                            "autoUpgradeMinorVersion": true,
                            "publisher": "Microsoft.Azure.Extensions",
                            "type": "CustomScript",
                            "typeHandlerVersion": "2.1"
                        }
                    }
                ]
            }
        },
        "provisioningState": "Succeeded",
        "overprovision": false,
        "doNotRunExtensionsOnOverprovisionedVMs": false,
        "uniqueId": "9af6d670-2987-479e-9c0a-7375d77aa313"
    }
}

To update the VMSS via the SDK, we are using the following code snippet:

public VirtualMachineScaleSet getLinuxEnterpriseRunnerVMSS() {
        return getAzureResourceManager().virtualMachineScaleSets()
                .getByResourceGroup(resourceGroup, linuxEnterpriseRunnerVMSSName);
}

protected void scaleUpLinuxRunnerVMSS(VirtualMachineScaleSet vmss) {
        
        // calculate new instance count 
        // [....] 
        vmss.update()
                .withCapacity(newVmInstanceCount)
                .apply();

        LOG.info("Scaled up VMSS {} to {} instances.", vmss.name(), newVmInstanceCount);
}

@Test
void scaleUpLinuxVMSS() {
        VirtualMachineScaleSet targetVMSS = githubRunnerService.getLinuxEnterpriseRunnerVMSS();
        githubRunnerService.scaleUpLinuxRunnerVMSS(targetVMSS);
}