Makes KFServing supports batch inference

/kind feature

Describe the solution you’d like

Makes KFServing supports batch inference.

It could take the following form:

BatchPredict POST /v1/models/<model_name>:batch_predict Request: {instances_path, predictions_path} Response: {}

[wronk]

I’m also looking to do batched inference with kubeflow, so +1 for this feature and thanks @yuzisun for those resources. Our use case is running inference over very large sets (10s to 100s of millions) of satellite images. On-demand or event triggering would also be helpful for us as the frequency can vary quite a bit by project (but likely between once a week to once every 3 months).

[radcheb]

As i understand, the ask is similar to Sagemaker batch transform?

Yes it’s a bit similar but should be more efficient. Sagemaker batch transform is using regular inference endpoints to batch inference with HTTP as transfer protocol. It’s not efficient for big datasets.