400 Request contains an invalid argument

Environment details

• OS type and version: Mac OSX 12.0.1
• Python Version: 3.8.11
• PIP version: 21.3.1
• google-cloud-platform version: 1.7.0

Steps to reproduce

1. Clone repo
2. Make predictions using sample located in python-aiplatform/samples/snippets/prediction_service/predict_custom_trained_model_sample.py.

Code example

from typing import Dict

from google.cloud import aiplatform
from google.protobuf import json_format
from google.protobuf.struct_pb2 import Value


def predict_custom_trained_model_sample(
    instance_dict: Dict,
    project: str = "xxx",
    endpoint_id: str = "xxx",
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.PredictionServiceClient(client_options=client_options)
    # The format of each instance should conform to the deployed model's prediction input schema.
    instance = json_format.ParseDict(instance_dict, Value())
    instances = [instance]
    parameters_dict = {}
    parameters = json_format.ParseDict(parameters_dict, Value())
    endpoint = client.endpoint_path(
        project=project, location=location, endpoint=endpoint_id
    )
    response = client.predict(
        endpoint=endpoint, instances=instances, parameters=parameters, timeout=10.0
    )
    print("response")
    print(" deployed_model_id:", response.deployed_model_id)
    # The predictions are a google.protobuf.Value representation of the model's predictions.
    predictions = response.predictions
    for prediction in predictions:
        print(" prediction:", dict(prediction))


if __name__ == "__main__":
    import json

    with open("payload.json", "r") as fp:
        data = json.loads(fp.read())

    predict_custom_trained_model_sample(instance_dict=data)

Stack trace

Traceback (most recent call last):
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Request contains an invalid argument."
        debug_error_string = "{"created":"@1636407165.720013000","description":"Error received from peer ipv6:[2607:f8b0:4005:809::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Request contains an invalid argument.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "b.py", line 45, in <module>
    predict_custom_trained_model_sample(instance_dict=data)
  File "b.py", line 28, in predict_custom_trained_model_sample
    response = client.predict(
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 473, in predict
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/google/api_core/timeout.py", line 102, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/taigaishida/.pyenv/versions/3.8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.

The error doesn’t occur every time I make a prediction, I would say in my limited testing that this error occurs %50 of the time.

[CapoD]

@TonyTheTaiga I can confirm, that we observe the same problem. This occurs only with gRPC. We deployed a quick hack that uses HTTP API instead which works for the moment.

[vinnysenthil]

Confirmed the issue is resolved. Unfortunately, I do not have additional details to share here.