Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

v2 compiler has data passing limitations

See original GitHub issue

Environment

kfp 1.6.2 kfp-pipeline-spec 0.1.7 kfp-server-api 1.5.0

Steps to reproduce

Use the upload_to_explicit_url component in a pipeline (picking a version that includes the latest v2 updates):

upload_op = load_component_from_url("https://raw.githubusercontent.com/kubeflow/pipelines/961b17fa6844e1d79e5d3686bb557d830d7b5a95/components/google-cloud/storage/upload_to_explicit_uri/component.yaml")
upload_task = upload_op(eval_task.output, results)  # pylint: disable=not-callable

As the Data input is untyped, it is treated as an artifact, and compiling produces this error:

TypeError: Passing PipelineParam "run-predictions-Results" with type "String" (as "Parameter") to component input "Data" with type "None" (as "Artifact") is incompatible. Please fix the type of the component input.

The fix appears straightforward, to include the String type on the Data input as well as the GCS path input and output.

Expected result

Pipeline compiles successfully.

Materials and Reference

https://raw.githubusercontent.com/kubeflow/pipelines/961b17fa6844e1d79e5d3686bb557d830d7b5a95/components/google-cloud/storage/upload_to_explicit_uri/component.yaml

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:11 (6 by maintainers)

Top GitHub Comments

1reaction

chensuncommented, Jun 19, 2021

I’ll take a look, cheers!

For reference, my version that uses a local component with this change does compile, and works as expected. I imagine you’d run into trouble if you wanted to upload an artifact rather than a string, but as most of my pipelines are being ported from v1 they don’t use artifact types (at least not intentionally: a lot of previously untyped parameters started being handled that way).

Hi @jackwhelpton, as we spoke last time on this issue, I believe your use case is unblocked by adding/updating the type. We will have a new doc with the detailed migration guide, and we will also update our components and samples to be compatible with v2 (https://github.com/kubeflow/pipelines/issues/5801).

If there are no other blockers, do you think we can close this issue? Thanks!

1reaction

Ark-kuncommented, May 25, 2021

The fix appears straightforward, to include the String type on the Data

This won’t work. The v2 compiler imposes severe limitations on the String type. The pipeline will still not compile and also you’ll lose the ability to upload any binary data like models.

A fix for this v2 compiler issue was proposed and implemented in https://github.com/kubeflow/pipelines/pull/5478 Please take a look.

Top Results From Across the Web

Chapter 3: System Limits and Programming Restrictions

The COBOL Compiler and the intermediate code that it produces are environment independent. The limits associated with the Compiler are environment independent ...

Single pass, Two pass, and Multi pass Compilers

Single pass compiler is one that processes the input exactly once, so going directly from lexical analysis to code generator, and then going ......

Optimize Options (Using the GNU Compiler Collection (GCC))

If loop unrolling is active, two passes are performed and the second is scheduled ... By default, GCC limits the size of functions...

CUDA Compiler Driver NVCC - NVIDIA Documentation Center

A value of 0 is allowed, and indicates that no limit should be enforced. This value is also passed to the host compiler...

One‐pass compilation at its limits — A modula‐2 compiler for ...

Abstract We present conceptual and implementation aspects of a one-pass Modula-2 compiler that was developed by Xerox PARC's future Dragon workstation.