question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

pytorch sample does not work

See original GitHub issue

/kind bug

What steps did you take and what happened: I’ve installed kfserving 0.1.0 on my minikube (none driver, and using NodePort for istio-ingress), and try to deploy the sample pytorch with below steps:

cd docs/samples/pytorch
kubectl apply -f pytorch.yaml

The pytorch-cifar10 kfservice could be created successfully, while the model is failed to work. Actually, sklearn and tensorflow are working on the same environment.

kubectl get kfservice
NAME              URL   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
flowers-sample          100                                15m
pytorch-cifar10                                            18h
sklearn-iris            100                                18h

Check below for more error output about it.

What did you expect to happen: The sample for pytorch should work.

Anything else you would like to add:

kubectl describe kfservice pytorch-cifar10
Name:         pytorch-cifar10
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"serving.kubeflow.org/v1alpha1","kind":"KFService","metadata":{"annotations":{},"name":"pytorch-cifar10","namespace":"defaul...
API Version:  serving.kubeflow.org/v1alpha1
Kind:         KFService
Metadata:
  Creation Timestamp:  2019-08-20T16:34:52Z
  Generation:          3
  Resource Version:    23304
  Self Link:           /apis/serving.kubeflow.org/v1alpha1/namespaces/default/kfservices/pytorch-cifar10
  UID:                 301078c2-2df2-4eff-a2d0-526ba41bca4a
Spec:
  Default:
    Pytorch:
      Model Class Name:  Net
      Model Uri:         gs://kfserving-samples/models/pytorch/cifar10/
      Resources:
        Requests:
          Cpu:          1
          Memory:       2Gi
      Runtime Version:  latest
Status:
  Canary:
  Conditions:
    Last Transition Time:  2019-08-20T16:36:54Z
    Message:               Revision "pytorch-cifar10-default-xqxtf" failed with message: Container failed with: INFO:root:Copying contents of /mnt/models to local
INFO:root:Copying contents of /mnt/models to local
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch-py37/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/pytorch-py37/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/pytorchserver/pytorchserver/__main__.py", line 38, in <module>
    model.load()
  File "/workspace/pytorchserver/pytorchserver/model.py", line 46, in load
    model_class = getattr(importlib.import_module(modulename), model_class_name)
  File "/opt/conda/envs/pytorch-py37/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
  File "<frozen importlib._bootstrap>", line 942, in _sanity_check
ValueError: Empty module name
.
    Reason:                RevisionFailed
    Status:                False
    Type:                  DefaultPredictorReady
    Last Transition Time:  2019-08-20T16:36:54Z
    Message:               Configuration "pytorch-cifar10-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Status:                False
    Type:                  Ready
    Last Transition Time:  2019-08-20T16:36:54Z
    Message:               Configuration "pytorch-cifar10-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Status:                False
    Type:                  RoutesReady
  Default:
    Name:  pytorch-cifar10-default-xqxtf
Events:    <none>

Environment:

  • Istio Version: v1.1.7
  • Knative Version: v0.8.0
  • KFServing Version: v0.1.0
  • Kubeflow version: N/A
  • Minikube version: v1.3.1
  • Kubernetes version: (use kubectl version):
kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server 7.6 (Maipo)

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:11 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
jinchihecommented, Aug 27, 2019

Used the last v0.1.2 docker image for pytorch, seems that’s OK.

By default, the kfserving get the latest image, but the latest image is not last one, list one is v0.1.2. Seems need to update the image tag.

0reactions
k8s-ci-robotcommented, Sep 28, 2019

@yuzisun: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Read more comments on GitHub >

github_iconTop Results From Across the Web

WightedRandomSampler does not sample uniformly
I am facing some problem with WeightedRandomSampler since I want to sample uniformly data among several labels. A really short description of the...
Read more >
pytorch sample does not work · Issue #296 · kserve ... - GitHub
The pytorch-cifar10 kfservice could be created successfully, while the model is failed to work. Actually, sklearn and tensorflow are working on ...
Read more >
DataLoader not randomly sampling in PyTorch - Stack Overflow
My model is only looking at the same single image (indexed '0') each time (batch size is 1...although nothing changes with different batch...
Read more >
Using PyTorch + NumPy? You're making a mistake.
Moreover, you won't have these issues if you sample random numbers using PyTorch (for example, torch.randint ) or Python's built-in random ...
Read more >
A detailed example of data loaders with PyTorch
We have to keep in mind that in some cases, even the most state-of-the-art configuration won't have enough memory space to process the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found