gRPC RESOURCE_EXHAUSTED w. "Received message larger than max" with bumped `max_{response, request}_size` config.

Context

• torchserve version: torchserve:0.4.1-gpu
• torch-model-archiver version: 0.4.2
• torch version: 1.9.0
• torchvision version [if any]: N/A
• torchtext version [if any]: N/A
• torchaudio version [if any]: N/A
• java version: see docker image
• Operating System and version: see docker image

Your Environment

• Installed using source? [yes/no]: no
• Are you planning to deploy it using docker container? [yes/no]: yes
• Is it a CPU or GPU environment?: GPU
• Using a default/custom handler? [If possible upload/share custom handler/model]: subclassing VisionHandler (more details below)
• What kind of model is it e.g. vision, text, audio?: vision
• Are you planning to use local models from model-store or public url being used e.g. from S3 bucket etc.? [If public url then provide link.]: local
• Provide config.properties, logs [ts.log] and parameters used for model registration/update APIs: enable_envvars_config=true in config.properties
• Link to your project [if any]: N/A

Expected Behavior

By setting the max_response_size config parameter I should be able to send messages over gRPC smaller or equal to the parameter value.

Current Behavior

I am querying torchserve using gRPC and on the client side I am getting the following error:

<_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (5649472 vs. 4194304)"
debug_error_string = "{"created":"@1630054530.266403986","description":"Received message larger than max (5649472 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":206,"grpc_status":8}"

I am using the torchserve:0.4.1-gpu docker image. I should be able to bump the max_response_size config parameter and hence send messages smaller than or equal to that size? Any idea how I can solve this?

Some additional information on my settings: I am using the following environment variables for the server

environment:
      - TS_CONFIG_FILE=/config/config.properties
      - TS_MAX_RESPONSE_SIZE=10000000
      - TS_MAX_REQUEST_SIZE=10000000

and the config.properties has the enable_envvars_config flag set to true

enable_envvars_config=true

In the official documentation the default size should be 6553500, however the error message provided above hints that the default size is 4194304.

Here are the logs upon startup of the torchserve service

torchserve_1  | Torchserve version: 0.4.1
torchserve_1  | TS Home: /usr/local/lib/python3.6/dist-packages
torchserve_1  | Current directory: /home/model-server
torchserve_1  | Temp directory: /home/model-server/tmp
torchserve_1  | Number of GPUs: 1
torchserve_1  | Number of CPUs: 16
torchserve_1  | Max heap size: 7990 M
torchserve_1  | Python executable: /usr/bin/python3
torchserve_1  | Config file: /config/config.properties
torchserve_1  | Inference address: http://127.0.0.1:8080
torchserve_1  | Management address: http://127.0.0.1:8081
torchserve_1  | Metrics address: http://127.0.0.1:8082
torchserve_1  | Model Store: /model-store
torchserve_1  | Initial Models: model.mar
torchserve_1  | Log dir: /home/model-server/logs
torchserve_1  | Metrics dir: /home/model-server/logs
torchserve_1  | Netty threads: 0
torchserve_1  | Netty client threads: 0
torchserve_1  | Default workers per model: 1
torchserve_1  | Blacklist Regex: N/A
torchserve_1  | Maximum Response Size: 10000000
torchserve_1  | Maximum Request Size: 10000000
torchserve_1  | Prefer direct buffer: false
torchserve_1  | Allowed Urls: [file://.*|http(s)?://.*]
torchserve_1  | Custom python dependency for model allowed: false
torchserve_1  | Metrics report format: prometheus
torchserve_1  | Enable metrics API: true
torchserve_1  | Workflow Store: /model-store
torchserve_1  | Model config: N/A

[jagadeeshi2i]

@hampusrosvall Looks fine with the configurations and from the torchserve logs, it seems the request size and response size are set to 10000000. Can you share the steps, so i can try to reproduce the issue.

[anishchhaparwal]

Making the following changes in torchserve_grpc_client.py solved the issue:

def get_inference_stub():
    options= [('grpc.max_receive_message_length',200*1024*1024)]
    channel = grpc.insecure_channel("localhost:7070", options=options)
    stub = inference_pb2_grpc.InferenceAPIsServiceStub(channel)
    return stub