pytorch sample does not work
See original GitHub issue/kind bug
What steps did you take and what happened:
I’ve installed kfserving 0.1.0 on my minikube (none driver, and using NodePort for istio-ingress), and try to deploy the sample pytorch
with below steps:
cd docs/samples/pytorch
kubectl apply -f pytorch.yaml
The pytorch-cifar10
kfservice could be created successfully, while the model is failed to work. Actually, sklearn and tensorflow are working on the same environment.
kubectl get kfservice
NAME URL DEFAULT TRAFFIC CANARY TRAFFIC AGE
flowers-sample 100 15m
pytorch-cifar10 18h
sklearn-iris 100 18h
Check below for more error output about it.
What did you expect to happen:
The sample for pytorch
should work.
Anything else you would like to add:
kubectl describe kfservice pytorch-cifar10
Name: pytorch-cifar10
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"serving.kubeflow.org/v1alpha1","kind":"KFService","metadata":{"annotations":{},"name":"pytorch-cifar10","namespace":"defaul...
API Version: serving.kubeflow.org/v1alpha1
Kind: KFService
Metadata:
Creation Timestamp: 2019-08-20T16:34:52Z
Generation: 3
Resource Version: 23304
Self Link: /apis/serving.kubeflow.org/v1alpha1/namespaces/default/kfservices/pytorch-cifar10
UID: 301078c2-2df2-4eff-a2d0-526ba41bca4a
Spec:
Default:
Pytorch:
Model Class Name: Net
Model Uri: gs://kfserving-samples/models/pytorch/cifar10/
Resources:
Requests:
Cpu: 1
Memory: 2Gi
Runtime Version: latest
Status:
Canary:
Conditions:
Last Transition Time: 2019-08-20T16:36:54Z
Message: Revision "pytorch-cifar10-default-xqxtf" failed with message: Container failed with: INFO:root:Copying contents of /mnt/models to local
INFO:root:Copying contents of /mnt/models to local
Traceback (most recent call last):
File "/opt/conda/envs/pytorch-py37/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/envs/pytorch-py37/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/workspace/pytorchserver/pytorchserver/__main__.py", line 38, in <module>
model.load()
File "/workspace/pytorchserver/pytorchserver/model.py", line 46, in load
model_class = getattr(importlib.import_module(modulename), model_class_name)
File "/opt/conda/envs/pytorch-py37/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
File "<frozen importlib._bootstrap>", line 942, in _sanity_check
ValueError: Empty module name
.
Reason: RevisionFailed
Status: False
Type: DefaultPredictorReady
Last Transition Time: 2019-08-20T16:36:54Z
Message: Configuration "pytorch-cifar10-default" does not have any ready Revision.
Reason: RevisionMissing
Status: False
Type: Ready
Last Transition Time: 2019-08-20T16:36:54Z
Message: Configuration "pytorch-cifar10-default" does not have any ready Revision.
Reason: RevisionMissing
Status: False
Type: RoutesReady
Default:
Name: pytorch-cifar10-default-xqxtf
Events: <none>
Environment:
- Istio Version: v1.1.7
- Knative Version: v0.8.0
- KFServing Version: v0.1.0
- Kubeflow version: N/A
- Minikube version: v1.3.1
- Kubernetes version: (use
kubectl version
):
kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
- OS (e.g. from
/etc/os-release
): Red Hat Enterprise Linux Server 7.6 (Maipo)
Issue Analytics
- State:
- Created 4 years ago
- Comments:11 (9 by maintainers)
Top Results From Across the Web
WightedRandomSampler does not sample uniformly
I am facing some problem with WeightedRandomSampler since I want to sample uniformly data among several labels. A really short description of the...
Read more >pytorch sample does not work · Issue #296 · kserve ... - GitHub
The pytorch-cifar10 kfservice could be created successfully, while the model is failed to work. Actually, sklearn and tensorflow are working on ...
Read more >DataLoader not randomly sampling in PyTorch - Stack Overflow
My model is only looking at the same single image (indexed '0') each time (batch size is 1...although nothing changes with different batch...
Read more >Using PyTorch + NumPy? You're making a mistake.
Moreover, you won't have these issues if you sample random numbers using PyTorch (for example, torch.randint ) or Python's built-in random ...
Read more >A detailed example of data loaders with PyTorch
We have to keep in mind that in some cases, even the most state-of-the-art configuration won't have enough memory space to process the...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Used the last v0.1.2 docker image for pytorch, seems that’s OK.
By default, the kfserving get the
latest
image, but thelatest
image is not last one, list one is v0.1.2. Seems need to update the image tag.@yuzisun: Closing this issue.
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.