kfserving setup does not work with s3 example
See original GitHub issue/kind bug
What steps did you take and what happened: [A clear and concise description of what the bug is.]
while running the aws-s3 example from the samples , StorageInitializer init container fails to initialize.
I cant even see the injection of env variables happening from the secret .
The error is : Traceback (most recent call last): File “/storage-initializer/scripts/initializer-entrypoint”, line 14, in <module> kfserving.Storage.download(src_uri, dest_path) File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 50, in download Storage._download_s3(uri, out_dir) File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 65, in _download_s3 client = Storage._create_minio_client() File “/usr/local/lib/python3.7/site-packages/kfserving/storage.py”, line 217, in _create_minio_client secure=use_ssl) File “/usr/local/lib/python3.7/site-packages/minio/api.py”, line 157, in init is_valid_endpoint(endpoint) File “/usr/local/lib/python3.7/site-packages/minio/helpers.py”, line 313, in is_valid_endpoint raise InvalidEndpointError(‘Hostname cannot be empty.’) minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
What did you expect to happen: The InferenceService starts properly.
Anything else you would like to add:
secret details
kind: Secret
metadata:
name: kfs-serving-secret
annotations:
serving.kubeflow.org/s3-endpoint: s3-eu-central-1.amazonaws.com
serving.kubeflow.org/s3-usehttps: "1"
serving.kubeflow.org/s3-verifyssl: "1"
serving.kubeflow.org/s3-region: eu-central-1
type: Opaque
data:
awsAccessKeyID: access-key-id
awsSecretAccessKey: awsSecretAccessKey
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kfs-serving-sa
secrets:
- name: kfs-serving-secret
InferenceService Definition :
apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
name: "mnist-s3"
spec:
default:
predictor:
serviceAccountName: kfs-serving-sa
tensorflow:
storageUri: "s3://bucket-name/mnist"
Pod Yaml :
k get pod mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/target: "1"
internal.serving.kubeflow.org/storage-initializer-sourceuri: s3://'nucket-name'/mnist
serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
creationTimestamp: "2020-08-06T05:07:45Z"
generateName: mnist-s3-predictor-default-sxrfw-deployment-684549d589-
labels:
app: mnist-s3-predictor-default-sxrfw
component: predictor
endpoint: default
model: mnist-s3
pod-template-hash: 684549d589
service.istio.io/canonical-name: mnist-s3-predictor-default
service.istio.io/canonical-revision: mnist-s3-predictor-default-sxrfw
serving.knative.dev/configuration: mnist-s3-predictor-default
serving.knative.dev/configurationGeneration: "1"
serving.knative.dev/revision: mnist-s3-predictor-default-sxrfw
serving.knative.dev/revisionUID: 390c94a1-f313-492b-99e5-931b0096f819
serving.knative.dev/service: mnist-s3-predictor-default
serving.kubeflow.org/inferenceservice: mnist-s3
name: mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: mnist-s3-predictor-default-sxrfw-deployment-684549d589
uid: 802abeab-b602-429c-89f2-46be2714cb43
resourceVersion: "994032"
selfLink: /api/v1/namespaces/default/pods/mnist-s3-predictor-default-sxrfw-deployment-684549d589-qq5r9
uid: 6012dd2c-690f-4661-bdf9-786379851aa8
spec:
containers:
- args:
- --port=9000
- --rest_api_port=8080
- --model_name=mnist-s3
- --model_base_path=/mnt/models
command:
- /usr/bin/tensorflow_model_server
env:
- name: PORT
value: "8080"
- name: K_REVISION
value: mnist-s3-predictor-default-sxrfw
- name: K_CONFIGURATION
value: mnist-s3-predictor-default
- name: K_SERVICE
value: mnist-s3-predictor-default
- name: K_INTERNAL_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: K_INTERNAL_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
httpGet:
path: /wait-for-drain
port: 8022
scheme: HTTP
livenessProbe:
failureThreshold: 3
httpGet:
httpHeaders:
- name: K-Kubelet-Probe
value: queue
path: /v1/models/mnist-s3
port: 8012
scheme: HTTP
initialDelaySeconds: 600
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: kfserving-container
ports:
- containerPort: 8080
name: user-port
protocol: TCP
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /var/log
name: knative-var-log
subPathExpr: $(K_INTERNAL_POD_NAMESPACE)_$(K_INTERNAL_POD_NAME)_kfserving-container
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kfs-serving-sa-token-kphtb
readOnly: true
- mountPath: /mnt/models
name: kfserving-provision-location
readOnly: true
- env:
- name: SERVING_NAMESPACE
value: default
- name: SERVING_SERVICE
value: mnist-s3-predictor-default
- name: SERVING_CONFIGURATION
value: mnist-s3-predictor-default
- name: SERVING_REVISION
value: mnist-s3-predictor-default-sxrfw
- name: QUEUE_SERVING_PORT
value: "8012"
- name: CONTAINER_CONCURRENCY
value: "0"
- name: REVISION_TIMEOUT_SECONDS
value: "60"
- name: SERVING_POD
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SERVING_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVING_LOGGING_CONFIG
value: |-
{
"level": "info",
"development": false,
"outputPaths": ["stdout"],
"errorOutputPaths": ["stderr"],
"encoding": "json",
"encoderConfig": {
"timeKey": "ts",
"levelKey": "level",
"nameKey": "logger",
"callerKey": "caller",
"messageKey": "msg",
"stacktraceKey": "stacktrace",
"lineEnding": "",
"levelEncoder": "",
"timeEncoder": "iso8601",
"durationEncoder": "",
"callerEncoder": ""
}
}
- name: SERVING_LOGGING_LEVEL
- name: SERVING_REQUEST_LOG_TEMPLATE
- name: SERVING_REQUEST_METRICS_BACKEND
value: prometheus
- name: TRACING_CONFIG_BACKEND
value: none
- name: TRACING_CONFIG_ZIPKIN_ENDPOINT
- name: TRACING_CONFIG_STACKDRIVER_PROJECT_ID
- name: TRACING_CONFIG_DEBUG
value: "false"
- name: TRACING_CONFIG_SAMPLE_RATE
value: "0.1"
- name: USER_PORT
value: "8080"
- name: SYSTEM_NAMESPACE
value: knative-serving
- name: METRICS_DOMAIN
value: knative.dev/internal/serving
- name: DOWNWARD_API_LABELS_PATH
value: /etc/podinfo/labels
- name: SERVING_READINESS_PROBE
value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
- name: ENABLE_PROFILING
value: "false"
- name: SERVING_ENABLE_PROBE_REQUEST_LOG
value: "false"
image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
imagePullPolicy: IfNotPresent
name: queue-proxy
ports:
- containerPort: 8022
name: http-queueadm
protocol: TCP
- containerPort: 9090
name: http-autometric
protocol: TCP
- containerPort: 9091
name: http-usermetric
protocol: TCP
- containerPort: 8012
name: queue-port
protocol: TCP
readinessProbe:
exec:
command:
- /ko-app/queue
- -probe-period
- "0"
failureThreshold: 3
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 10
resources:
requests:
cpu: 25m
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kfs-serving-sa-token-kphtb
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- args:
- s3://bucket-name/mnist
- /mnt/models
image: gcr.io/kfserving/storage-initializer:v0.3.0
imagePullPolicy: IfNotPresent
name: storage-initializer
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /mnt/models
name: kfserving-provision-location
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kfs-serving-sa-token-kphtb
readOnly: true
nodeName: docker-desktop
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: kfs-serving-sa
serviceAccountName: kfs-serving-sa
terminationGracePeriodSeconds: 60
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: knative-var-log
- name: kfs-serving-sa-token-kphtb
secret:
defaultMode: 420
secretName: kfs-serving-sa-token-kphtb
- emptyDir: {}
name: kfserving-provision-location
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-08-06T05:07:45Z"
message: 'containers with incomplete status: [storage-initializer]'
reason: ContainersNotInitialized
status: "False"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-08-06T05:07:45Z"
message: 'containers with unready status: [kfserving-container queue-proxy]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-08-06T05:07:45Z"
message: 'containers with unready status: [kfserving-container queue-proxy]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-08-06T05:07:45Z"
status: "True"
type: PodScheduled
containerStatuses:
- image: index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
imageID: ""
lastState: {}
name: kfserving-container
ready: false
restartCount: 0
started: false
state:
waiting:
reason: PodInitializing
- image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
imageID: ""
lastState: {}
name: queue-proxy
ready: false
restartCount: 0
started: false
state:
waiting:
reason: PodInitializing
hostIP: 192.168.65.3
initContainerStatuses:
- containerID: docker://313d9511bfc6e5b072b54c6cbecf315e0447737b12cf166fd18e7ada83d4e7af
image: gcr.io/kfserving/storage-initializer:v0.3.0
imageID: docker-pullable://gcr.io/kfserving/storage-initializer@sha256:7900509fe1fbe4a14fa4692bfb105e2a1d086c78390d5901ab9cc7d9e9a54853
lastState:
terminated:
containerID: docker://539df5136983a5d98571b9f7f9edaefe5bcaf3c4b70eaaf3a038b7e7190fbc64
exitCode: 1
finishedAt: "2020-08-06T05:29:05Z"
message: |
[I 200806 05:29:05 initializer-entrypoint:13] Initializing, args: src_uri [s3://bucket-name/mnist] dest_path[ [/mnt/models]
[I 200806 05:29:05 storage:35] Copying contents of s3://bucket-name/mnist to local
Traceback (most recent call last):
File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
kfserving.Storage.download(src_uri, dest_path)
File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
Storage._download_s3(uri, out_dir)
File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
client = Storage._create_minio_client()
File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
secure=use_ssl)
File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 157, in __init__
is_valid_endpoint(endpoint)
File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 313, in is_valid_endpoint
raise InvalidEndpointError('Hostname cannot be empty.')
minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
reason: Error
startedAt: "2020-08-06T05:29:04Z"
name: storage-initializer
ready: false
restartCount: 10
state:
terminated:
containerID: docker://313d9511bfc6e5b072b54c6cbecf315e0447737b12cf166fd18e7ada83d4e7af
exitCode: 1
finishedAt: "2020-08-06T05:34:10Z"
message: |
[I 200806 05:34:09 initializer-entrypoint:13] Initializing, args: src_uri [s3://hbucket-name/mnist] dest_path[ [/mnt/models]
[I 200806 05:34:09 storage:35] Copying contents of s3://bucket-name/mnist to local
Traceback (most recent call last):
File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
kfserving.Storage.download(src_uri, dest_path)
File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
Storage._download_s3(uri, out_dir)
File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
client = Storage._create_minio_client()
File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
secure=use_ssl)
File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 157, in __init__
is_valid_endpoint(endpoint)
File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 313, in is_valid_endpoint
raise InvalidEndpointError('Hostname cannot be empty.')
minio.error.InvalidEndpointError: InvalidEndpointError: message: Hostname cannot be empty.
reason: Error
startedAt: "2020-08-06T05:34:08Z"
phase: Pending
podIP: 10.1.0.201
podIPs:
- ip: 10.1.0.201
qosClass: Burstable
startTime: "2020-08-06T05:07:45Z"
Environment:
- Istio Version: 1.6.5
- Knative Version: 0.16.0
- KFServing Version: 0.3.0
- Storage_initializer “gcr.io/kfserving/storage-initializer:v0.3.0”,
- Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube version: using Docker for desktop 2.3.0.3
- Kubernetes version: (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}```
- OS (e.g. from `/etc/os-release`):
Issue Analytics
- State:
- Created 3 years ago
- Comments:9 (4 by maintainers)
Top GitHub Comments
@navneet1075 In kfserving 0.4.0,
awsAccessKeyID
andawsSecretAccessKey
have been replaced byAWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
.@amercier-twt Thanks for your reply, I have submitted a Pull Request.