VolumeOp was not able to create PVC

What steps did you take:

A simple pipeline with one vol_op and one simple step that mounts the pvc created.

# minimal example

import kfp

@kfp.dsl.pipeline(
    name="data download and upload",
    description="Data IO test"
)
def volume_pipeline():
    # shared vol
    
    vop = kfp.dsl.VolumeOp(
        name="volume_creation",
        resource_name="sharedpvc",
        size="5Gi",
        modes=["RWO"]
    )
    
    # mount the vol
    
    simple_task = kfp.dsl.ContainerOp(
        name="simple task",
        image="bash",
        arguments=[
            "echo",
            "hello",
            ">/data/hello.text"
        ]
    ).add_pvolumes({
        "/data": vop.volume
    })
    
# run the pipeline

client = kfp.Client()

client.create_run_from_pipeline_func(volume_pipeline, arguments={})

What happened:

The VolumeOp was not able to create the PVC, therefore the depending task complains about not finding the PVC. kubectl get pvc -n kubeflow | grep sharedpvc didn’t return any results.

What did you expect to happen:

The VolumeOp shall create a PVC named sharedpvc.

Environment:

How did you deploy Kubeflow Pipelines (KFP)? Deploying Kubeflow Pipelines on a local kind cluster

KFP version: 1.2.0

KFP SDK version: 1.4.0

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

The log of the VolumneOp indicates

This step output is taken from cache.

I was trying to prevent it from using the cache but didn’t succeed. The manifest from the VolumneOp

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: '{{workflow.name}}-sharedpvc' spec: accessModes: - RWO resources: requests: storage: 5Gi 

The log of the depending task indicates

This step is in Pending state with this message: Unschedulable: persistentvolumeclaim "{{tasks.volume-creation.outputs.parameters.volume-creation-name}}" not found

Storage class used

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"standard"},"provisioner":"rancher.io/local-path","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2020-12-29T06:31:17Z"
  name: standard
  resourceVersion: "195"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
  uid: 5dbc1bff-b488-4d3a-b45f-e710cf96a415
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

/kind bug

[elikatsis]

I’ll reopen this issue as it needs some fix apart from globally disabling the cache

/reopen

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.