Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Kubeflow / AI Platform Pipelines runtime context missing when output is taken from cache

See original GitHub issue

What steps did you take:

deploy pipeline with one component
run pipeline with one component (👍 works)
add another component
run the pipeline (this time the output is taken from cache) (👎 fails)

What happened:

The pipeline runs into the following error

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main
    execution_info = launcher.launch()
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 198, in launch
    self._exec_properties)
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 167, in _run_driver
    component_info=self._component_info)
  File "/opt/conda/lib/python3.7/site-packages/tfx/dsl/components/base/base_driver.py", line 270, in pre_execution
    driver_args, pipeline_info)
  File "/opt/conda/lib/python3.7/site-packages/tfx/dsl/components/base/base_driver.py", line 158, in resolve_input_artifacts
    producer_component_id=input_channel.producer_component_id)
  File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/metadata.py", line 948, in search_artifacts
    pipeline_info)
RuntimeError: Pipeline run context for PipelineInfo(pipeline_name: sentiment4, pipeline_root: gs://sascha-playground-doit-kubeflowpipelines-default/sentiment4, run_id: sentiment4-qnknl) does not exist

Assume the second component doesn’t find the cached data because the component did not exist in the first run.

First run: 1run

Second run with additional component 2run

What did you expect to happen:

Pipeline run completes without errors

Environment:

AI Platform Pipelines

How did you deploy Kubeflow Pipelines (KFP)? AI Platform Pipelines

KFP version: https://github.com/kubeflow/pipelines/commit/d79071c0bef19442483abc101769a0d893e72f42

KFP SDK version: no pip in AI Platform Pipelines

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

Issue Analytics

State:
Created 3 years ago
Comments:24 (14 by maintainers)

Top GitHub Comments

1reaction

Bobgycommented, Jul 14, 2021

@AlirezaSadeghi did you read the docs: https://www.kubeflow.org/docs/distributions/gke/pipelines/upgrade/?

1reaction

SaschaHeyercommented, Apr 7, 2021

Hi @Bobgy here is the example you can use https://github.com/SaschaHeyer/Sentiment-Analysis-TFX

Top Results From Across the Web

Caching | Kubeflow

Kubeflow Pipelines caching provides step-level output caching. And caching is enabled by default for all pipelines submitted through the KFP ...

Amazon Elastic File System (EFS) – AWS Machine Learning Blog

Kubeflow on AWS helps build a highly available and robust ML platform. ... The pipeline evaluates the conditional runtime variables and decides between ......

tfx Changelog - pyup.io

Output artifacts from multiple invocations of the same component are given ... TFX CLI now supports runtime parameter on Kubeflow, Vertex, and Airflow....

Source - GitHub

Fixes [\#8366](https://github.com/kubeflow/pipelines/issues/8366) ... fix(frontend) Missing parameter in the input/output tab of Run details side panel ...

A Tour of End-to-End Machine Learning Platforms - KDnuggets

You can think of things such as versioning, experiment management, run-time statistics, data lineage tracking for training, test, and validation ...