question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

GCP Vertex AI, deploying model to existing endpoint

See original GitHub issue

Feature Area

/area sdk /area components

I currently have a simple pipeline that deploys to Vertex AI that takes an existing model container image from Artifact Registry, uploads it to the Vertex AI model store, creates an endpoint and deploys the model to that endpoint. So far so good.
Now say I have a new revision of that model and I want to deploy that to the same endpoint with a 50/50 traffic split between the old and new version. This is achievable relatively simply in the Vertex AI gui but how do I do this in a kubeflow pipeline?
How am I able to get either an existing model (already uploaded) or an already created endpoint to pass to the ModelDeployOp?
The following code snippet does not work as the returned endpoint type doesnt plug into the operation.

def pipeline(
    project: str = PROJECT_ID,
    model_display_name: str = MODEL_DISPLAY_NAME,
    serving_container_image_uri: str = IMAGE_URI,
):
    train_task = print_op("No training to be done here!")
    
    model_upload_op = gcc_aip.ModelUploadOp(
        project=project, 
        location=REGION,
        display_name=model_display_name,
        # artifact_uri=WORKING_DIR,
        serving_container_image_uri=serving_container_image_uri,
        serving_container_ports=[{"containerPort": 8000}],
        serving_container_predict_route="/hello_world",
        serving_container_health_route="/health",    
     )
    
    endpoints = aip.Endpoint.list(
                filter=f"display_name={ENDPOINT_NAME}", order_by="create_time"
            )
    existing_endpoint = endpoints[0]
    existing_endpoint = existing_endpoint
    

    model_deploy_op = gcc_aip.ModelDeployOp(
        endpoint=existing_endpoint,
        model=model_upload_op.outputs["model"],
        deployed_model_display_name=model_display_name,
        dedicated_resources_machine_type="n1-standard-4",
        dedicated_resources_min_replica_count=1,
        dedicated_resources_max_replica_count=1,
        traffic_split=traffic_split,
    )

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:1
  • Comments:9

github_iconTop GitHub Comments

1reaction
clausagerskovcommented, Mar 1, 2022

@lc-billyfung it seems your code was close enough for me to get it working with some minor modifications. scrolling through the other issues in the repo i found a mention of this command “.ignore_type()”. using that, with a small correction to your str split and the pipeline deploys:

from google_cloud_pipeline_components import aiplatform as gcc_aip
import kfp
@kfp.dsl.pipeline(name="multimodel-test")
def pipeline(
    project: str = PROJECT_ID,
    model_display_name: str = MODEL_DISPLAY_NAME,
    serving_container_image_uri: str = IMAGE_URI,
):
    train_task = print_op("No training to be done here!")
    
    model_upload_op = gcc_aip.ModelUploadOp(
        project=project, 
        location=REGION,
        display_name=model_display_name,
        # artifact_uri=WORKING_DIR,
        serving_container_image_uri=serving_container_image_uri,
        serving_container_ports=[{"containerPort": 8000}],
        serving_container_predict_route="/hello_world",
        serving_container_health_route="/health",    
     )
    
    custom_deploy_op = custom_deploy_to_endpoint(
        model=model_upload_op.outputs['model'].ignore_type(),
        endpoint_display_name=ENDPOINT_NAME,
        project=PROJECT_ID
    )
    custom_deploy_op.after(model_upload_op)

@component(base_image="python:3.9", packages_to_install=['google-cloud-aiplatform'])
def custom_deploy_to_endpoint(
    model: Input[Model],
    endpoint_display_name: str,
    project: str,
):
    import sys
    from google.cloud import aiplatform

    aiplatform.init(project=project)
    target_endpoint = None
    for endpoint in aiplatform.Endpoint.list(order_by="update_time desc"):
        if endpoint.display_name == endpoint_display_name:
            target_endpoint = endpoint
    
    if target_endpoint is None:
        target_endpoint = aiplatform.Endpoint.create(
            project=project,
            display_name=endpoint_display_name,
        )
        
    model_id = model.uri.split('models/')[1]
    target_model = aiplatform.Model(model_id)
    target_endpoint.deploy(
        model=target_model,
        min_replica_count=1,
        max_replica_count=1,
        traffic_percentage=50,
        machine_type='n1-standard-4',
    )
1reaction
lc-billyfungcommented, Feb 24, 2022

I’m doing this by using a custom component that checks for existing endpoints based on display_name and then deploys the model to it if found.

aiplatform.init(project=project)
target_endpoint = None
for endpoint in aiplatform.Endpoint.list(order_by="update_time desc"):
    if endpoint.display_name == endpoint_display_name:
        target_endpoint = endpoint

if target_endpoint is None:
    target_endpoint = aiplatform.Endpoint.create(
        project=project,
        display_name=endpoint_display_name,
    )

  target_model = aiplatform.Model(model_id)
    target_endpoint.deploy(
        model=target_model,
        min_replica_count=1,
        max_replica_count=1,
        traffic_percentage=100,
        machine_type='n1-standard-4',
    )
Read more comments on GitHub >

github_iconTop Results From Across the Web

Overview of getting predictions on Vertex AI | Google Cloud
Deploying a model associates physical resources with the model so it can serve online predictions with low latency.
Read more >
Deploy your own custom ML model on Vertex AI using GCP ...
Vertex AI model deployment contains two main concepts: Model and Endpoint. Model is a kind of package that contains all components needed to...
Read more >
Serving Machine Learning models with Google Vertex AI
Deploy model · You have to upload your model to the Vertex AI Model Registry. · Create a Vertex AI Endpoint (once) ·...
Read more >
Deploying Machine Learning models with Vertex AI (GCP)
A model must be deployed to an endpoint before they can serve online predictions. When you deploy a model, GCP associates physical resources...
Read more >
Deploying ViT on Vertex AI - Hugging Face
1. The first step in the workflow is to upload the SavedModel to Vertex AI's model registry: · 2. Then you need to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found