Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

not able start kfserving with s3

See original GitHub issue

/kind bug

Running kfserving with kubeflow installed on minkube for s3 tensorflow storage model but awaiting at PodInitializing

kubectl get inferenceservice -n kfserving-test-2
NAME             URL                                                                           READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
effidet-s3                                                                                     False                                      83m
effidet-s3-1                                                                                   False                                      7m30s
flowers-sample   http://flowers-sample.kfserving-test-2.example.com/v1/models/flowers-sample   True    100                                109m
sklearn-iris     http://sklearn-iris.kfserving-test-2.example.com/v1/models/sklearn-iris       True    100                                17h

deployment.apps/effidet-s3-1-predictor-default-khtjq-deployment     0/0     0            0           6m45s
deployment.apps/effidet-s3-1-predictor-default-nc6kn-deployment     0/0     0            0           6m45s
replicaset.apps/effidet-s3-1-predictor-default-khtjq-deployment-55fc98844f    0         0         0       6m45s
replicaset.apps/effidet-s3-1-predictor-default-nc6kn-deployment-6dcfb7bf56    0         0         0       6m45s
revision.serving.knative.dev/effidet-s3-1-predictor-default-khtjq     effidet-s3-1-predictor-default     effidet-s3-1-predictor-default-khtjq     1            False   PodInitializing
revision.serving.knative.dev/effidet-s3-1-predictor-default-nc6kn     effidet-s3-1-predictor-default     effidet-s3-1-predictor-default-nc6kn     2            False   PodInitializing
configuration.serving.knative.dev/effidet-s3-1-predictor-default     effidet-s3-1-predictor-default-nc6kn                                              False   RevisionFailed
service.serving.knative.dev/effidet-s3-1-predictor-default     http://effidet-s3-1-predictor-default.kfserving-test-2.example.com     effidet-s3-1-predictor-default-nc6kn                                              False   RevisionMissing
route.serving.knative.dev/effidet-s3-1-predictor-default     http://effidet-s3-1-predictor-default.kfserving-test-2.example.com     False   RevisionMissing

      s3://bucket_name/savd_model
      /mnt/models
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 10 Jul 2020 08:04:58 +0000
      Finished:     Fri, 10 Jul 2020 08:05:00 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 10 Jul 2020 08:04:41 +0000
      Finished:     Fri, 10 Jul 2020 08:04:43 +0000
    Ready:          False
    Restart Count:  2

Deployment file :

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "effidet-s3-1"
spec:
  default:
    predictor:
      serviceAccountName: sa
      tensorflow:
        storageUri: "s3://bucket/saved_model"
---
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    serving.kubeflow.org/s3-endpoint: s3.amazonaws.com.
    serving.kubeflow.org/s3-usehttps: "1" 
    serving.kubeflow.org/s3-verifyssl: "1"
    serving.kubeflow.org/s3-region: us-east-1
type: Opaque
stringData:
  awsAccessKeyID: xxxx=
  awsSecretAccessKey: xxx==
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
  - name: mysecret

Environment:

Kubeflow version: v1.0.2
Minikube version: v1.8.0
Kubernetes version: v1.17.3
OS (e.g. from /etc/os-release):

Issue Analytics

State:
Created 3 years ago
Comments:12 (4 by maintainers)

Top GitHub Comments

2reactions

aishwary09commented, Nov 2, 2020

" I have replicated the same with minio client. its authentication keys" can anyone suggest what this means, in final comment when @tiru1930 found the solution

2reactions

tiru1930commented, Jul 10, 2020

issue with keys

Top Results From Across the Web

not able start kfserving with s3 · Issue #929 - GitHub

Running kfserving with kubeflow installed on minkube for s3 tensorflow storage model but awaiting at PodInitializing.

Debugging guide - KServe Documentation Website

You deployed an InferenceService to KServe, but it is not in ready state. Go through this step by step guide to understand what...

Kfserving -- Error When Defining storageUri - Stack Overflow

KFServing is injecting a second container in the pod of the predictor, SKLearn in your case, which is called storage_initializer .

KFserving Storage Initializer (Deprecated) — seldon-core ...

When kfserving/storage-initializer is used modeluri supports the following four object storage providers: Google Cloud Storage (using gs:// ). S3-compatible ...

S3 Model Serving - CoreWeave

Deploying a model hosted in Amazon S3 is straightforward. ... A transformer is not required and the predictor can be queried directly with...