avoid declaring two times same function (and params) in vertex-pipeline with py-functions

context: With vertex-pipeline + py-functions We can use a base_image (docker) with our py library pre-installed. Therefore the lib functions are available to import inside the pipeline.

problem: Assuming we want to use one of this functions (without any change) in the pipeline, is a little verbose to declare with @component the same function again, which is corresponding paramater types. Would be cleaner if you could just declare it once. And the way the code works, wouldn’t be so much of a change.

solution proposal: assuming our function is in our library: from kfp_test.test import test then we have the option

1. clean but verbose

@component(base_image=base_image)
    def test(x: int) -> int:
        kwargs = locals()
        from kfp_test.test import test
        return test(**kwargs)`

1. not clean but not verbose

from inspect import getsource as getsource2

# deal with line 93 in component_factory.py: func_code = inspect.getsource(func)
def getsource(func):
    if 'func_code' in func.__dict__.keys():
        return func.func_code
    else:

        return getsource2(func)
from kfp.v2 import compiler
(...)
from kfp_test.test import test
 func_code = f'    def {inspect.signature(test)}:\n        kwargs = locals()\n        from kfp_test.test import test\n        return test(**kwargs)\n'
    test.func_code = func_code
    test = create_component_from_func(func=test, base_image=base_image)

if we had a function in kfp like create_component_from_func_code that handled option 2 natively (create a wrapper string automatically from the func and ingestit in the create_component() without declaring it) , then we would have a non-verbose, clean-way to use functions from the library inside the pipeline

[iuiu34]

yep… Anway, if someone out there is in a similar situation: Wants to convert py-funcs in a custom py-package into kfp-py-func components. But without changing the code (i.e. without putting the imports inside, etc.) Here you have an util create-wrapper-components that will create a components.py file automatically link

Requirements:

• your functions should have type hint for input and output args (and only kfp-py-func available types)
• your base_image for the components and the image/pc where you run the create-wrapper-components should have the same your_custom_package version

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.