question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Configuration parsing error when reusing an input in a composite solid.

See original GitHub issue

If I have understood composite solids correctly, then I would expect to be able to run the following example where I reuse the arg to calculate_result in two of the inner solids. However, I get an error:

dagster.core.errors.DagsterInvalidConfigError: Pipeline “my_pipeline” config errors: Error 1: Missing required field “calculate_result” at path root:solids Available Fields: “[‘calculate_result’, ‘noop’]”.

from dagster import execute_pipeline, pipeline, lambda_solid, composite_solid


@lambda_solid
def noop(arg):
    return arg


@lambda_solid
def add_one(arg):
    return arg + 1


@lambda_solid
def adder(first, second):
    return first + second


@composite_solid
def calculate_result(arg):
    left = add_one.alias('add_one_left')(arg)
    right = add_one.alias('add_one_right')(arg)
    return adder(left, right)


@pipeline
def my_pipeline():
    return calculate_result(noop())


if __name__ == '__main__':
    execute_pipeline(
        my_pipeline,
        environment_dict={
            'solids': {
                'noop': {'inputs': {'arg': {'value': 1}}},
                # 'calculate_result': {'solids': {'add_one_left': {'inputs': {'arg': {'value': 10}}}}},
            }
        }
    )

If I uncomment the line in the environment_dict to fill the value for one of the solids inside, then the pipeline will run. Alternatively, if I change calculate_sum and my_pipeline so that I pass the same argument twice to calculate_sum then the pipeline will run:

@composite_solid
def calculate_result(arg1, arg2):
    left = add_one.alias('add_one_left')(arg1)
    right = add_one.alias('add_one_right')(arg2)
    return adder(left, right)


@pipeline
def my_pipeline():
    a = noop()
    return calculate_result(a, a)

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
alangenfeldcommented, Jul 11, 2019
0reactions
jack-pacecommented, Jul 10, 2019

Thanks, and yes, this is another real-world use case 😄

Read more comments on GitHub >

github_iconTop Results From Across the Web

JSON Parser Error at Offset - TechDocs
This article provides information for Error: JSON Parser Error at Offset.
Read more >
Cannot setup WireGuard VPN - Server Fault
Line unrecognized: `Address=10.0.0.1/24,fd86:ea04:1115::1/64' Configuration parsing error. What you can also check:.
Read more >
Reply To: “Reusing” Fluent Simulation Settings
Just to be clear, I am not wondering about how to make sure only the inlet condition is changed but rather, how I...
Read more >
KNIME Components Guide
In this guide, we explain how to create components and metanodes, how to create composite views and configuration dialogs for components, and how...
Read more >
Bison 3.8.1 - GNU.org
The fact that the parser state is reusable even after an error simplifies reuse. For example, a calculator application which parses each input...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found