question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

kfserving setup does not work with s3 example

See original GitHub issue

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

while running the aws-s3 example from the samples , StorageInitializer init container fails to initialize.

I cant even see the injection of env variables happening from the secret .

The error is : Traceback (most recent call last): File “/storage-initializer/scripts/initializer-entrypoint”, line 14, in <module> kfserving.Storage.download(src_uri, dest_path) File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 50, in download Storage._download_s3(uri, out_dir) File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 65, in _download_s3 client = Storage._create_minio_client() File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 217, in _create_minio_client secure=use_ssl) File “/usr/local/lib/python3.7/site-packages/minio/api.py”, line 157, in init is_valid_endpoint(endpoint) File “/usr/local/lib/python3.7/site-packages/minio/helpers.py”, line 313, in is_valid_endpoint raise InvalidEndpointError(‘Hostname cannot be empty.’) minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.

What did you expect to happen: The InferenceService starts properly.

Anything else you would like to add:

secret details

kind: Secret
metadata:
  name: kfs-serving-secret
  annotations:
     serving.kubeflow.org/s3-endpoint: s3-eu-central-1.amazonaws.com
     serving.kubeflow.org/s3-usehttps: "1"
     serving.kubeflow.org/s3-verifyssl: "1"
     serving.kubeflow.org/s3-region: eu-central-1

type: Opaque
data:
  awsAccessKeyID: access-key-id
  awsSecretAccessKey: awsSecretAccessKey

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfs-serving-sa
secrets:
  - name: kfs-serving-secret

InferenceService Definition :

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "mnist-s3"
spec:
  default:
    predictor:
      serviceAccountName: kfs-serving-sa
      tensorflow:
        storageUri: "s3://bucket-name/mnist"

Pod Yaml :

k get pod mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
    autoscaling.knative.dev/minScale: "1"
    autoscaling.knative.dev/target: "1"
    internal.serving.kubeflow.org/storage-initializer-sourceuri: s3://'nucket-name'/mnist
    serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
  creationTimestamp: "2020-08-06T05:07:45Z"
  generateName: mnist-s3-predictor-default-sxrfw-deployment-684549d589-
  labels:
    app: mnist-s3-predictor-default-sxrfw
    component: predictor
    endpoint: default
    model: mnist-s3
    pod-template-hash: 684549d589
    service.istio.io/canonical-name: mnist-s3-predictor-default
    service.istio.io/canonical-revision: mnist-s3-predictor-default-sxrfw
    serving.knative.dev/configuration: mnist-s3-predictor-default
    serving.knative.dev/configurationGeneration: "1"
    serving.knative.dev/revision: mnist-s3-predictor-default-sxrfw
    serving.knative.dev/revisionUID: 390c94a1-f313-492b-99e5-931b0096f819
    serving.knative.dev/service: mnist-s3-predictor-default
    serving.kubeflow.org/inferenceservice: mnist-s3
  name: mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: mnist-s3-predictor-default-sxrfw-deployment-684549d589
    uid: 802abeab-b602-429c-89f2-46be2714cb43
  resourceVersion: "994032"
  selfLink: /api/v1/namespaces/default/pods/mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9
  uid: 6012dd2c-690f-4661-bdf9-786379851aa8
spec:
  containers:
  - args:
    - --port=9000
    - --rest_api_port=8080
    - --model_name=mnist-s3
    - --model_base_path=/mnt/models
    command:
    - /usr/bin/tensorflow_model_server
    env:
    - name: PORT
      value: "8080"
    - name: K_REVISION
      value: mnist-s3-predictor-default-sxrfw
    - name: K_CONFIGURATION
      value: mnist-s3-predictor-default
    - name: K_SERVICE
      value: mnist-s3-predictor-default
    - name: K_INTERNAL_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: K_INTERNAL_POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        httpGet:
          path: /wait-for-drain
          port: 8022
          scheme: HTTP
    livenessProbe:
      failureThreshold: 3
      httpGet:
        httpHeaders:
        - name: K-Kubelet-Probe
          value: queue
        path: /v1/models/mnist-s3
        port: 8012
        scheme: HTTP
      initialDelaySeconds: 600
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: kfserving-container
    ports:
    - containerPort: 8080
      name: user-port
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /var/log
      name: knative-var-log
      subPathExpr: $(K_INTERNAL_POD_NAMESPACE)_$(K_INTERNAL_POD_NAME)_kfserving-container
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kfs-serving-sa-token-kphtb
      readOnly: true
    - mountPath: /mnt/models
      name: kfserving-provision-location
      readOnly: true
  - env:
    - name: SERVING_NAMESPACE
      value: default
    - name: SERVING_SERVICE
      value: mnist-s3-predictor-default
    - name: SERVING_CONFIGURATION
      value: mnist-s3-predictor-default
    - name: SERVING_REVISION
      value: mnist-s3-predictor-default-sxrfw
    - name: QUEUE_SERVING_PORT
      value: "8012"
    - name: CONTAINER_CONCURRENCY
      value: "0"
    - name: REVISION_TIMEOUT_SECONDS
      value: "60"
    - name: SERVING_POD
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: SERVING_POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: SERVING_LOGGING_CONFIG
      value: |-
        {
          "level": "info",
          "development": false,
          "outputPaths": ["stdout"],
          "errorOutputPaths": ["stderr"],
          "encoding": "json",
          "encoderConfig": {
            "timeKey": "ts",
            "levelKey": "level",
            "nameKey": "logger",
            "callerKey": "caller",
            "messageKey": "msg",
            "stacktraceKey": "stacktrace",
            "lineEnding": "",
            "levelEncoder": "",
            "timeEncoder": "iso8601",
            "durationEncoder": "",
            "callerEncoder": ""
          }
        }
    - name: SERVING_LOGGING_LEVEL
    - name: SERVING_REQUEST_LOG_TEMPLATE
    - name: SERVING_REQUEST_METRICS_BACKEND
      value: prometheus
    - name: TRACING_CONFIG_BACKEND
      value: none
    - name: TRACING_CONFIG_ZIPKIN_ENDPOINT
    - name: TRACING_CONFIG_STACKDRIVER_PROJECT_ID
    - name: TRACING_CONFIG_DEBUG
      value: "false"
    - name: TRACING_CONFIG_SAMPLE_RATE
      value: "0.1"
    - name: USER_PORT
      value: "8080"
    - name: SYSTEM_NAMESPACE
      value: knative-serving
    - name: METRICS_DOMAIN
      value: knative.dev/internal/serving
    - name: DOWNWARD_API_LABELS_PATH
      value: /etc/podinfo/labels
    - name: SERVING_READINESS_PROBE
      value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
    - name: ENABLE_PROFILING
      value: "false"
    - name: SERVING_ENABLE_PROBE_REQUEST_LOG
      value: "false"
    image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
    imagePullPolicy: IfNotPresent
    name: queue-proxy
    ports:
    - containerPort: 8022
      name: http-queueadm
      protocol: TCP
    - containerPort: 9090
      name: http-autometric
      protocol: TCP
    - containerPort: 9091
      name: http-usermetric
      protocol: TCP
    - containerPort: 8012
      name: queue-port
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /ko-app/queue
        - -probe-period
        - "0"
      failureThreshold: 3
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 10
    resources:
      requests:
        cpu: 25m
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kfs-serving-sa-token-kphtb
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - s3://bucket-name/mnist
    - /mnt/models
    image: gcr.io/kfserving/storage-initializer:v0.3.0
    imagePullPolicy: IfNotPresent
    name: storage-initializer
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /mnt/models
      name: kfserving-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kfs-serving-sa-token-kphtb
      readOnly: true
  nodeName: docker-desktop
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: kfs-serving-sa
  serviceAccountName: kfs-serving-sa
  terminationGracePeriodSeconds: 60
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: knative-var-log
  - name: kfs-serving-sa-token-kphtb
    secret:
      defaultMode: 420
      secretName: kfs-serving-sa-token-kphtb
  - emptyDir: {}
    name: kfserving-provision-location
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    message: 'containers with incomplete status: [storage-initializer]'
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    message: 'containers with unready status: [kfserving-container queue-proxy]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    message: 'containers with unready status: [kfserving-container queue-proxy]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
    imageID: ""
    lastState: {}
    name: kfserving-container
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  - image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
    imageID: ""
    lastState: {}
    name: queue-proxy
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  hostIP: 192.168.65.3
  initContainerStatuses:
  - containerID: docker://313d9511bfc6e5b072b54c6cbecf315e0447737b12cf166fd18e7ada83d4e7af
    image: gcr.io/kfserving/storage-initializer:v0.3.0
    imageID: docker-pullable://gcr.io/kfserving/storage-initializer@sha256:7900509fe1fbe4a14fa4692bfb105e2a1d086c78390d5901ab9cc7d9e9a54853
    lastState:
      terminated:
        containerID: docker://539df5136983a5d98571b9f7f9edaefe5bcaf3c4b70eaaf3a038b7e7190fbc64
        exitCode: 1
        finishedAt: "2020-08-06T05:29:05Z"
        message: |
          [I 200806 05:29:05 initializer-entrypoint:13] Initializing, args: src_uri [s3://bucket-name/mnist] dest_path[ [/mnt/models]
          [I 200806 05:29:05 storage:35] Copying contents of s3://bucket-name/mnist to local
          Traceback (most recent call last):
            File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
              kfserving.Storage.download(src_uri, dest_path)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
              Storage._download_s3(uri, out_dir)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
              client = Storage._create_minio_client()
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
              secure=use_ssl)
            File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 157, in __init__
              is_valid_endpoint(endpoint)
            File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 313, in is_valid_endpoint
              raise InvalidEndpointError('Hostname cannot be empty.')
          minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
        reason: Error
        startedAt: "2020-08-06T05:29:04Z"
    name: storage-initializer
    ready: false
    restartCount: 10
    state:
      terminated:
        containerID: docker://313d9511bfc6e5b072b54c6cbecf315e0447737b12cf166fd18e7ada83d4e7af
        exitCode: 1
        finishedAt: "2020-08-06T05:34:10Z"
        message: |
          [I 200806 05:34:09 initializer-entrypoint:13] Initializing, args: src_uri [s3://hbucket-name/mnist] dest_path[ [/mnt/models]
          [I 200806 05:34:09 storage:35] Copying contents of s3://bucket-name/mnist to local
          Traceback (most recent call last):
            File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
              kfserving.Storage.download(src_uri, dest_path)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
              Storage._download_s3(uri, out_dir)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
              client = Storage._create_minio_client()
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
              secure=use_ssl)
            File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 157, in __init__
              is_valid_endpoint(endpoint)
            File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 313, in is_valid_endpoint
              raise InvalidEndpointError('Hostname cannot be empty.')
          minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
        reason: Error
        startedAt: "2020-08-06T05:34:08Z"
  phase: Pending
  podIP: 10.1.0.201
  podIPs:
  - ip: 10.1.0.201
  qosClass: Burstable
  startTime: "2020-08-06T05:07:45Z"

Environment:

  • Istio Version: 1.6.5
  • Knative Version: 0.16.0
  • KFServing Version: 0.3.0
  • Storage_initializer “gcr.io/kfserving/storage-initializer:v0.3.0”,
  • Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
  • Minikube version: using Docker for desktop 2.3.0.3
  • Kubernetes version: (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}```
- OS (e.g. from `/etc/os-release`):

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
AnyISalIncommented, Aug 24, 2020

@navneet1075 In kfserving 0.4.0, awsAccessKeyID and awsSecretAccessKey have been replaced by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

0reactions
AnyISalIncommented, Jul 10, 2021

@amercier-twt Thanks for your reply, I have submitted a Pull Request.

Read more comments on GitHub >

github_iconTop Results From Across the Web

kfserving setup does not work with s3 example #999 - GitHub
while running the aws-s3 example from the samples , StorageInitializer init container ... kfserving setup does not work with s3 example #999.
Read more >
KFserving Storage Initializer (Deprecated) — seldon-core ...
When kfserving/storage-initializer is used modeluri supports the following four object storage providers: Google Cloud Storage (using gs:// ). S3-compatible ...
Read more >
Tutorial: Configuring a static website on Amazon S3
This example walks you through the steps of hosting a website on Amazon S3. Topics. Step 1: Create a bucket; Step 2: Enable...
Read more >
Kfserving -- Error When Defining storageUri - Stack Overflow
KFServing is injecting a second container in the pod of the predictor, SKLearn in your case, which is called storage_initializer .
Read more >
MLflow Models — MLflow 2.0.1 documentation
For example, int -> long or int -> double conversions are ok, long -> double is not. If the types cannot be made...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found