Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TorchServe ignores batch config properties

See original GitHub issue

In my config.properties file, I have the lines:

batch_size=4
max_batch_delay=200

I started TorchServe with the command line:

torchserve --start --ts-config config.properties --models d161good=d161good.mar  --model-store model_store

When I query the status of the endpoint with curl http://127.0.0.1:8081/models/d161good, I get:

[
  {
    "modelName": "d161good",
    "modelVersion": "1.0",
    "modelUrl": "d161good.mar",
    "runtime": "python",
    "minWorkers": 12,
    "maxWorkers": 12,
    "batchSize": 1,
    "maxBatchDelay": 100,
    "loadedAtStartup": true,
...

Note the "batchSize" and "maxBatchDelay" entries.

Issue Analytics

State:
Created 4 years ago
Reactions:1
Comments:12 (7 by maintainers)

Top GitHub Comments

5reactions

punshrivcommented, Jun 1, 2021

@harshbafna If batchSize and max_batch_delay can only be configured only through management API what is the recommendation from Torchserve team to configure this when using multiple replicas in Kubernetes to load these values on container start/restart ?

0reactions

darkain84commented, Feb 24, 2022

@harshbafna If batchSize and max_batch_delay can only be configured only through management API what is the recommendation from Torchserve team to configure this when using multiple replicas in Kubernetes to load these values on container start/restart ?

I found an example for that in torchserve github. https://github.com/pytorch/serve/blob/master/kubernetes/EKS/config.properties.

I hope the above link will be helpful.