question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

String as an argument to the node in pipeline.

See original GitHub issue

I want to add feature engineering in the pipeline for every feature and for that I want to pass a string as an argument to the function inside the node but that’s not allowed in kedro yet and I get following error:- ValueError: Pipeline input(s) {"xxxx","yyyy"} not found in the DataCatalog So I want to know if there’s a way to use string as in argument inside node.

Thanks in advance.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

6reactions
WaylonWalkercommented, Sep 26, 2020

Currently kedro only supports passing parameters or catalog entries to nodes. The way that you would pass other options in would be with a partial or a lambda. I most often use lambdas, you can look up how to do it with functools.partial if you would rather use that.

def my_func(data, param):
   data[feature] = param
   return data

node(
   func= lambda data: my_func(data, 'xxx')
   ...
)

node(
   func= lambda data: my_func(data, 'yyy')
   ...
)

I often generate nodes dynamically, be careful to bind your inputs to your lambdas if you create them in a loop. More on that in this article https://waylonwalker.com/blog/bind-dynamic-lambdas.

0reactions
sansagaracommented, Sep 26, 2020

Thats a nice suggestion, i must say.

Read more comments on GitHub >

github_iconTop Results From Across the Web

String as an argument to the node in pipeline. #213 - GitHub
I want to add feature engineering in the pipeline for every feature and for that I want to pass a string as an...
Read more >
Pipeline Syntax - Jenkins
A string. Runtime arguments to pass to docker run . This option is valid for docker and dockerfile . Example 1. Docker Agent,...
Read more >
How To Use Parameters In Jenkins Declarative Pipeline
This blog post explains different ways to use parameters in Jenkins declarative pipeline including dynamic active choice parameters.
Read more >
Pass Jenkins build parameters to pipeline nodes
Each parameter becomes a variable of the Pipeline script binding. Therefore you can access them directly by name. In your example echo "$VAR_A" ......
Read more >
Using Declarative Pipeline syntax - CloudBees Documentation
A string. The label on which to run the Pipeline or individual stage . This option is valid for node , docker and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found