Selection for partitioned in an AssetGroup
See original GitHub issueSummary
Hi everyone,
I want to define multiple asset branches in a single AssetGroup
and I would like each branch to be partitioned with a different start_date
. So, I created several jobs using a selection
. But obviously the asset selection for partitioning is done on the AssetGroup and not on the selection. What generates this exception : When an assets job contains multiple partitions assets, they must have the same partitions definitions, but asset '["branch_1_upstream"]' and asset '["branch_2_upstream"]' have different partitions definitions
.
Is there a solution to solve this problem? Thank you in advance
Reproduction
def make_asset_group(branch, date):
daily = DailyPartitionsDefinition(start_date=date)
@asset(
name = f'{branch}_upstream',
partitions_def=daily
)
def asset_upstream():
...
@asset(
name = f'{branch}_downstream',
ins={"upstream": AssetIn(asset_key=AssetKey(f'{branch}_upstream'),
partitions_def=daily
)
def asset_downstream(upstream):
...
return [asset_downstream, asset_upstream]
assets_branch = make_asset_group(branch, date) for branch, date in {branch_1 : date_1, branch_2 : date_2}.items()
asset_group = AssetGroup(
assets=[asset for assets in assets_branch for asset in assets]
)
job_1 = asset_group.build_job(
selection = [branch_1_upstream, branch_1_downstream]
)
job_2 = asset_group.build_job(
selection = [branch_2_upstream, branch_2_downstream]
)
@repository
def repository():
return [asset_group, job1, job2]
Dagit UI/UX Issue Screenshots
Additional Info about Your Environment
Message from the maintainers:
Impacted by this bug? Give it a 👍. We factor engagement into prioritization.
Issue Analytics
- State:
- Created a year ago
- Reactions:2
- Comments:5 (5 by maintainers)
Top Results From Across the Web
Partitioned assets and jobs - Dagster Docs
In the top bar, you can select from the list of all available partitions. Within the config editor, the config for the selected...
Read more >Turning the data pipeline inside out | Georg Heiler
The desired range of partitions can be selected manually in the UI or by specifying the desired matching wildcard.
Read more >One Identity Safeguard for Privileged Passwords 7.0.1 LTS
Navigate to Asset Management > Partitions. In Partitions, select a partition and click Set as Default from the toolbar. Setting a default profile....
Read more >19.5 Partition Selection
19.5 Partition Selection. MySQL 5.6 supports explicit selection of partitions and subpartitions that, when executing a statement, should be checked for rows ...
Read more >Raw scans
Select RAW partition and click Next to scan for lost files. ... Findings Report are based on the same (1) asset group and...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@jburnich - yesterday’s release included a change that enables having assets with different partitions definitions within the same group. That turned out to make more sense than enabling multiple AssetGroups within the same repository.
I believe that should address the issue you raised here, so I’m going to close this. Feel free to reopen if not.
Thanks for your quick return. Yes, that’s a good idea. I believe that if a repository can contain multiple AssetGroup where each AssetGroup has its own partitioning and scheduler, this would probably solve our problem. We look forward to testing this new feature.