storage-initialiser fails to copy model from S3

Upon using a Secret and Service account that look like this:

apiVersion: v1
metadata:
  name: kfserving-aws-secret
  namespace: {namespace}
  annotations:
    serving.kubeflow.org/s3-endpoint: s3.eu-east-1.amazonaws.com
    serving.kubeflow.org/s3-usehttps: "1"
    serving.kubeflow.org/s3-verifyssl: "1"
    serving.kubeflow.org/s3-region: eu-east-1
data:
  AWS_ACCESS_KEY_ID: {access_key_base64}
  AWS_SECRET_ACCESS_KEY: {secret_key_base64}
kind: Secret
type: Opaque
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfserving-sa
  namespace: {namespace}
secrets:
  - name: kfserving-aws-secret  

…and an InferenceService like this:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: {model_name}
  namespace: {namespace}
spec:
  default:
    predictor:
      serviceAccountName: kfserving-sa
      tensorflow:
        storageUri: {model_base_path}    

…the model fails to serve, with the storage-initialiser pod throwing the following error:

[I 200226 14:09:42 initializer-entrypoint:13] Initializing, args: src_uri [s3://at-kubeflow-training/tf-job/ksc-ebce76a1] dest_path[ [/mnt/models]
[I 200226 14:09:42 storage:35] Copying contents of s3://at-kubeflow-training/tf-job/ksc-ebce76a1 to local
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kfserving.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
    Storage._download_s3(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
    client = Storage._create_minio_client()
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
    secure=use_ssl)
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 150, in __init__
    is_valid_endpoint(endpoint)
  File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 301, in is_valid_endpoint
    if hostname[-1] == '.':
IndexError: string index out of range

This was using Kubeflow 1.0.0RC4, KFServing 0.2.2 and KNative 0.11.1

[yuzisun]

I think the name is copied from original kubeflow s3 example since we want to keep consistent, I just checked it is changed there too, so feel free to raise a PR to fix the default then. Thanks for brought this up.

[marcindulak]

https://www.kubeflow.org/docs/aws/aws-e2e/ shows an example of the kind: Secret which uses

data:
  # echo -ne "AKIAxxx" | base64
  awsAccessKeyID: QUtJQVhxxxVXVjQ=
  awsSecretAccessKey: QzR0UnxxxVNOd0NQQQ==

I’ve just used the original AWS variable naming successfully (at least in terms of storage-initializer not failing with IndexError: string index out of range) with minio.

Let me post a typical minio setup (https://github.com/kubeflow/kfserving/tree/master/docs/samples/s3) in case someone needs it

apiVersion: v1
kind: Secret
metadata:
  name: kfserving-minio-secret
  namespace: anonymous
  annotations:
    serving.kubeflow.org/s3-endpoint: minio-service.kubeflow:9000
    serving.kubeflow.org/s3-usehttps: "0"
    serving.kubeflow.org/s3-verifyssl: "0"
    serving.kubeflow.org/s3-region: us-east-1
type: Opaque
stringData:
  awsAccessKeyID: minio
  awsSecretAccessKey: minio123
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfserving-sa
  namespace: anonymous
secrets:
  - name: kfserving-minio-secret