question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Selection for partitioned in an AssetGroup

See original GitHub issue

Summary

Hi everyone, I want to define multiple asset branches in a single AssetGroup and I would like each branch to be partitioned with a different start_date. So, I created several jobs using a selection. But obviously the asset selection for partitioning is done on the AssetGroup and not on the selection. What generates this exception : When an assets job contains multiple partitions assets, they must have the same partitions definitions, but asset '["branch_1_upstream"]' and asset '["branch_2_upstream"]' have different partitions definitions. Is there a solution to solve this problem? Thank you in advance

Reproduction

def make_asset_group(branch, date):
    daily = DailyPartitionsDefinition(start_date=date)
    
    @asset(
        name = f'{branch}_upstream',
        partitions_def=daily
    )
    def asset_upstream():
        ...

    @asset(
        name = f'{branch}_downstream',
        ins={"upstream": AssetIn(asset_key=AssetKey(f'{branch}_upstream'),
        partitions_def=daily
    )
    def asset_downstream(upstream):
        ...

    return [asset_downstream, asset_upstream]


assets_branch = make_asset_group(branch, date) for branch, date in {branch_1 : date_1, branch_2 : date_2}.items()

asset_group = AssetGroup(
  assets=[asset for assets in assets_branch for asset in assets]
)

job_1 = asset_group.build_job(
    selection = [branch_1_upstream, branch_1_downstream]
)
job_2 = asset_group.build_job(
    selection = [branch_2_upstream, branch_2_downstream]
)


@repository
def repository():
    return [asset_group, job1, job2]

Dagit UI/UX Issue Screenshots

Additional Info about Your Environment


Message from the maintainers:

Impacted by this bug? Give it a 👍. We factor engagement into prioritization.

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:2
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
sryzacommented, Apr 15, 2022

@jburnich - yesterday’s release included a change that enables having assets with different partitions definitions within the same group. That turned out to make more sense than enabling multiple AssetGroups within the same repository.

I believe that should address the issue you raised here, so I’m going to close this. Feel free to reopen if not.

0reactions
jburnichcommented, Apr 8, 2022

Thanks for your quick return. Yes, that’s a good idea. I believe that if a repository can contain multiple AssetGroup where each AssetGroup has its own partitioning and scheduler, this would probably solve our problem. We look forward to testing this new feature.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Partitioned assets and jobs - Dagster Docs
In the top bar, you can select from the list of all available partitions. Within the config editor, the config for the selected...
Read more >
Turning the data pipeline inside out | Georg Heiler
The desired range of partitions can be selected manually in the UI or by specifying the desired matching wildcard.
Read more >
One Identity Safeguard for Privileged Passwords 7.0.1 LTS
Navigate to Asset Management > Partitions. In Partitions, select a partition and click Set as Default from the toolbar. Setting a default profile....
Read more >
19.5 Partition Selection
19.5 Partition Selection. MySQL 5.6 supports explicit selection of partitions and subpartitions that, when executing a statement, should be checked for rows ...
Read more >
Raw scans
Select RAW partition and click Next to scan for lost files. ... Findings Report are based on the same (1) asset group and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found