kfserving setup does not work with s3 example

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

while running the aws-s3 example from the samples , StorageInitializer init container fails to initialize.

I cant even see the injection of env variables happening from the secret .

The error is : Traceback (most recent call last): File “/storage-initializer/scripts/initializer-entrypoint”, line 14, in <module> kfserving.Storage.download(src_uri, dest_path) File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 50, in download Storage._download_s3(uri, out_dir) File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 65, in _download_s3 client = Storage._create_minio_client() File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 217, in _create_minio_client secure=use_ssl) File “/usr/local/lib/python3.7/site-packages/minio/api.py”, line 157, in init is_valid_endpoint(endpoint) File “/usr/local/lib/python3.7/site-packages/minio/helpers.py”, line 313, in is_valid_endpoint raise InvalidEndpointError(‘Hostname cannot be empty.’) minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.

What did you expect to happen: The InferenceService starts properly.

Anything else you would like to add:

secret details

kind: Secret
metadata:
  name: kfs-serving-secret
  annotations:
     serving.kubeflow.org/s3-endpoint: s3-eu-central-1.amazonaws.com
     serving.kubeflow.org/s3-usehttps: "1"
     serving.kubeflow.org/s3-verifyssl: "1"
     serving.kubeflow.org/s3-region: eu-central-1

type: Opaque
data:
  awsAccessKeyID: access-key-id
  awsSecretAccessKey: awsSecretAccessKey

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfs-serving-sa
secrets:
  - name: kfs-serving-secret

InferenceService Definition :

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "mnist-s3"
spec:
  default:
    predictor:
      serviceAccountName: kfs-serving-sa
      tensorflow:
        storageUri: "s3://bucket-name/mnist"

Pod Yaml :

k get pod mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
    autoscaling.knative.dev/minScale: "1"
    autoscaling.knative.dev/target: "1"
    internal.serving.kubeflow.org/storage-initializer-sourceuri: s3://'nucket-name'/mnist
    serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
  creationTimestamp: "2020-08-06T05:07:45Z"
  generateName: mnist-s3-predictor-default-sxrfw-deployment-684549d589-
  labels:
    app: mnist-s3-predictor-default-sxrfw
    component: predictor
    endpoint: default
    model: mnist-s3
    pod-template-hash: 684549d589
    service.istio.io/canonical-name: mnist-s3-predictor-default
    service.istio.io/canonical-revision: mnist-s3-predictor-default-sxrfw
    serving.knative.dev/configuration: mnist-s3-predictor-default
    serving.knative.dev/configurationGeneration: "1"
    serving.knative.dev/revision: mnist-s3-predictor-default-sxrfw
    serving.knative.dev/revisionUID: 390c94a1-f313-492b-99e5-931b0096f819
    serving.knative.dev/service: mnist-s3-predictor-default
    serving.kubeflow.org/inferenceservice: mnist-s3
  name: mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: mnist-s3-predictor-default-sxrfw-deployment-684549d589
    uid: 802abeab-b602-429c-89f2-46be2714cb43
  resourceVersion: "994032"
  selfLink: /api/v1/namespaces/default/pods/mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9
  uid: 6012dd2c-690f-4661-bdf9-786379851aa8
spec:
  containers:
  - args:
    - --port=9000
    - --rest_api_port=8080
    - --model_name=mnist-s3
    - --model_base_path=/mnt/models
    command:
    - /usr/bin/tensorflow_model_server
    env:
    - name: PORT
      value: "8080"
    - name: K_REVISION
      value: mnist-s3-predictor-default-sxrfw
    - name: K_CONFIGURATION
      value: mnist-s3-predictor-default
    - name: K_SERVICE
      value: mnist-s3-predictor-default
    - name: K_INTERNAL_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: K_INTERNAL_POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        httpGet:
          path: /wait-for-drain
          port: 8022
          scheme: HTTP
    livenessProbe:
      failureThreshold: 3
      httpGet:
        httpHeaders:
        - name: K-Kubelet-Probe
          value: queue
        path: /v1/models/mnist-s3
        port: 8012
        scheme: HTTP
      initialDelaySeconds: 600
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: kfserving-container
    ports:
    - containerPort: 8080
      name: user-port
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /var/log
      name: knative-var-log
      subPathExpr: $(K_INTERNAL_POD_NAMESPACE)_$(K_INTERNAL_POD_NAME)_kfserving-container
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kfs-serving-sa-token-kphtb
      readOnly: true
    - mountPath: /mnt/models
      name: kfserving-provision-location
      readOnly: true
  - env:
    - name: SERVING_NAMESPACE
      value: default
    - name: SERVING_SERVICE
      value: mnist-s3-predictor-default
    - name: SERVING_CONFIGURATION
      value: mnist-s3-predictor-default
    - name: SERVING_REVISION
      value: mnist-s3-predictor-default-sxrfw
    - name: QUEUE_SERVING_PORT
      value: "8012"
    - name: CONTAINER_CONCURRENCY
      value: "0"
    - name: REVISION_TIMEOUT_SECONDS
      value: "60"
    - name: SERVING_POD
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: SERVING_POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: SERVING_LOGGING_CONFIG
      value: |-
        {
          "level": "info",
          "development": false,
          "outputPaths": ["stdout"],
          "errorOutputPaths": ["stderr"],
          "encoding": "json",
          "encoderConfig": {
            "timeKey": "ts",
            "levelKey": "level",
            "nameKey": "logger",
            "callerKey": "caller",
            "messageKey": "msg",
            "stacktraceKey": "stacktrace",
            "lineEnding": "",
            "levelEncoder": "",
            "timeEncoder": "iso8601",
            "durationEncoder": "",
            "callerEncoder": ""
          }
        }
    - name: SERVING_LOGGING_LEVEL
    - name: SERVING_REQUEST_LOG_TEMPLATE
    - name: SERVING_REQUEST_METRICS_BACKEND
      value: prometheus
    - name: TRACING_CONFIG_BACKEND
      value: none
    - name: TRACING_CONFIG_ZIPKIN_ENDPOINT
    - name: TRACING_CONFIG_STACKDRIVER_PROJECT_ID
    - name: TRACING_CONFIG_DEBUG
      value: "false"
    - name: TRACING_CONFIG_SAMPLE_RATE
      value: "0.1"
    - name: USER_PORT
      value: "8080"
    - name: SYSTEM_NAMESPACE
      value: knative-serving
    - name: METRICS_DOMAIN
      value: knative.dev/internal/serving
    - name: DOWNWARD_API_LABELS_PATH
      value: /etc/podinfo/labels
    - name: SERVING_READINESS_PROBE
      value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
    - name: ENABLE_PROFILING
      value: "false"
    - name: SERVING_ENABLE_PROBE_REQUEST_LOG
      value: "false"
    image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
    imagePullPolicy: IfNotPresent
    name: queue-proxy
    ports:
    - containerPort: 8022
      name: http-queueadm
      protocol: TCP
    - containerPort: 9090
      name: http-autometric
      protocol: TCP
    - containerPort: 9091
      name: http-usermetric
      protocol: TCP
    - containerPort: 8012
      name: queue-port
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /ko-app/queue
        - -probe-period
        - "0"
      failureThreshold: 3
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 10
    resources:
      requests:
        cpu: 25m
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kfs-serving-sa-token-kphtb
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - s3://bucket-name/mnist
    - /mnt/models
    image: gcr.io/kfserving/storage-initializer:v0.3.0
    imagePullPolicy: IfNotPresent
    name: storage-initializer
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /mnt/models
      name: kfserving-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kfs-serving-sa-token-kphtb
      readOnly: true
  nodeName: docker-desktop
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: kfs-serving-sa
  serviceAccountName: kfs-serving-sa
  terminationGracePeriodSeconds: 60
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: knative-var-log
  - name: kfs-serving-sa-token-kphtb
    secret:
      defaultMode: 420
      secretName: kfs-serving-sa-token-kphtb
  - emptyDir: {}
    name: kfserving-provision-location
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    message: 'containers with incomplete status: [storage-initializer]'
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    message: 'containers with unready status: [kfserving-container queue-proxy]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    message: 'containers with unready status: [kfserving-container queue-proxy]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-08-06T05:07:45Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
    imageID: ""
    lastState: {}
    name: kfserving-container
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  - image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
    imageID: ""
    lastState: {}
    name: queue-proxy
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  hostIP: 192.168.65.3
  initContainerStatuses:
  - containerID: docker://313d9511bfc6e5b072b54c6cbecf315e0447737b12cf166fd18e7ada83d4e7af
    image: gcr.io/kfserving/storage-initializer:v0.3.0
    imageID: docker-pullable://gcr.io/kfserving/storage-initializer@sha256:7900509fe1fbe4a14fa4692bfb105e2a1d086c78390d5901ab9cc7d9e9a54853
    lastState:
      terminated:
        containerID: docker://539df5136983a5d98571b9f7f9edaefe5bcaf3c4b70eaaf3a038b7e7190fbc64
        exitCode: 1
        finishedAt: "2020-08-06T05:29:05Z"
        message: |
          [I 200806 05:29:05 initializer-entrypoint:13] Initializing, args: src_uri [s3://bucket-name/mnist] dest_path[ [/mnt/models]
          [I 200806 05:29:05 storage:35] Copying contents of s3://bucket-name/mnist to local
          Traceback (most recent call last):
            File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
              kfserving.Storage.download(src_uri, dest_path)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
              Storage._download_s3(uri, out_dir)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
              client = Storage._create_minio_client()
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
              secure=use_ssl)
            File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 157, in __init__
              is_valid_endpoint(endpoint)
            File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 313, in is_valid_endpoint
              raise InvalidEndpointError('Hostname cannot be empty.')
          minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
        reason: Error
        startedAt: "2020-08-06T05:29:04Z"
    name: storage-initializer
    ready: false
    restartCount: 10
    state:
      terminated:
        containerID: docker://313d9511bfc6e5b072b54c6cbecf315e0447737b12cf166fd18e7ada83d4e7af
        exitCode: 1
        finishedAt: "2020-08-06T05:34:10Z"
        message: |
          [I 200806 05:34:09 initializer-entrypoint:13] Initializing, args: src_uri [s3://hbucket-name/mnist] dest_path[ [/mnt/models]
          [I 200806 05:34:09 storage:35] Copying contents of s3://bucket-name/mnist to local
          Traceback (most recent call last):
            File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
              kfserving.Storage.download(src_uri, dest_path)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
              Storage._download_s3(uri, out_dir)
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
              client = Storage._create_minio_client()
            File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
              secure=use_ssl)
            File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 157, in __init__
              is_valid_endpoint(endpoint)
            File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 313, in is_valid_endpoint
              raise InvalidEndpointError('Hostname cannot be empty.')
          minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
        reason: Error
        startedAt: "2020-08-06T05:34:08Z"
  phase: Pending
  podIP: 10.1.0.201
  podIPs:
  - ip: 10.1.0.201
  qosClass: Burstable
  startTime: "2020-08-06T05:07:45Z"

Environment:

• Istio Version: 1.6.5
• Knative Version: 0.16.0
• KFServing Version: 0.3.0
• Storage_initializer “gcr.io/kfserving/storage-initializer:v0.3.0”,
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube version: using Docker for desktop 2.3.0.3
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}```
- OS (e.g. from `/etc/os-release`):

[AnyISalIn]

@navneet1075 In kfserving 0.4.0, awsAccessKeyID and awsSecretAccessKey have been replaced by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

[AnyISalIn]

@amercier-twt Thanks for your reply, I have submitted a Pull Request.