Sagemaker autoscaling not working using boto3 client

Describe the bug I want to programmatically configure auto-scaling in AWS Sagemaker. I am using the official boto3 documentation. First step is to register a scalable target. Second step is to define a scaling policy on the target. The first step works fine. Second step gets executed but it wont change the metric value .

Steps to reproduce

#Step - 1  -- This step works fine. Max and Min capacities are updated as passed in the arguments
        autoscaling.register_scalable_target(
            ServiceNamespace='sagemaker',
            ResourceId='endpoint/test-20450427-0648/variant/test-20450427-0648-model-kp-vrjfgs268gfljnhs5ra',
            ScalableDimension='sagemaker:variant:DesiredInstanceCount',
            MinCapacity=1,
            MaxCapacity=10,
            RoleARN='arn:aws:iam::654763378725:role/ml-sagemaker'
        )

#Step-2  -- This step is not working. 
        autoscaling.put_scaling_policy(
            PolicyName='InvocationsPerInstanceCustom',
            ServiceNamespace='sagemaker',
            ResourceId='endpoint/test-20450427-0648/variant/test-20450427-0648-model-kp-vrjfgs268gfljnhs5ra',
            ScalableDimension='sagemaker:variant:DesiredInstanceCount',
            PolicyType='TargetTrackingScaling',
            TargetTrackingScalingPolicyConfiguration={
                "TargetValue": 122.0,
                "PredefinedMetricSpecification": {"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"},
                'ScaleOutCooldown': 60,
                'ScaleInCooldown': 60
            }
        )

Expected behavior Step 2 should update the target value of metric SageMakerVariantInvocationsPerInstance to 122. But it remains empty when checked on AWS console . ScaleOutCooldown and ScaleInCooldown also should be set to 60 but they show default value 300. Debug logs No error message is repoted.

[pradoz]

@georgebakas

PolicyName="SagemakerEndpointInvocationScalingPolicy ", …there is a space here^

[stobrien89]

Hi @singh0777,

Glad you were able to get it working! I don’t think you did anything wrong here— it seems like, in theory, you should be able to apply custom policies to a target regardless of whether or not a ‘built-in’ policy has already been applied. This may be a bug with the Sagemaker console; I’ve asked the team to elaborate on what the expected behavior should be and am currently waiting on a response.

I have a hunch that your custom policy was actually being applied to the target, but it was not displaying in the console due to the name. I’ll let you know as soon as I have more information!