Amazon EKS example DAG raises warnning
See original GitHub issueBody
Importing package: airflow.providers.amazon.aws.example_dags
[2022-05-17 15:53:10,675] {eks.py:287} WARNING - The nodegroup_subnets should be List or string representing Python list and is {{ dag_run.conf['nodegroup_subnets'] }}. Defaulting to []
example for waning raising in the CI: https://github.com/apache/airflow/runs/6474027737?check_suite_focus=true#step:12:1010
We should fix it. The example DAG should not yield warnings.
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Issue Analytics
- State:
- Created a year ago
- Comments:5 (5 by maintainers)
Top Results From Across the Web
Amazon EKS troubleshooting - AWS Documentation
Nodes fail to join cluster ... There are a few common reasons that prevent nodes from joining the cluster: ... The node is...
Read more >Performance tuning for Apache Airflow on Amazon MWAA
Open the Environments page on the Amazon MWAA console. ... For example, if you specify a value of 30 , the DAG file...
Read more >Using Amazon MWAA with Amazon EKS - AWS Documentation
The following sample demonstrates how to use Amazon Managed Workflows for Apache Airflow (MWAA) with Amazon EKS.
Read more >Learn why your EKS pod is stuck in the ContainerCreating state
This error relates to the Amazon VPC admission controller webhook that's required on Amazon EKS clusters to run Windows workloads. The webhook ...
Read more >apache-airflow-providers-amazon Documentation
Fix Amazon EKS example DAG raises warning during Imports (#23849). Move string arg evals to 'execute()' in 'EksCreateClusterOperator' (#23877).
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
The issue is that
nodegroup_subnets
is a template field for the operator and the string evaluation on that arg is happening as part of the constructor. We’ve been slowly finding more and more of these instances and pushing them out of__init__()
, but template fields are not rendered (i.e. retain their stringified Jinja value) until just before theexecute()
method is called. So string evaluations on template fields should not happen until theexecute()
scope. This would also generally emit a warning ifnodegroup_subnets
was anXComArg
as well.Moving that evaluation section to the scope of the
execute()
method should do the trick and be functionally correct.Hi, I looked into the issue and was confused about what to do.
ast.literal_eval
is not able to process this string{{ dag_run.conf['nodegroup_subnets'] }}
I also looked in other files where literal_eval is used.DockerOperator
uses it but I’m getting errors there as well. Can you please help me with correct string format or do we need to make changes inEksCreateNodegroupOperator