question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(aws-mwaa): unable to use existing LogGroup via via mwaa.CfnEnvironment

See original GitHub issue

I am unable to use existing LogGroups via mwaa.CfnEnvironment. cdk deploy raised error

Reproduction Steps

class AirflowStack(core.Stack):

    def __init__(self, scope: core.Construct, construct_id: str, environment: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        scheduler_log = aws_logs.LogGroup(
                scope=self,
                id=f'airflow-cloudwatch-{environment}-Scheduler',
                log_group_name=f'/aws/airflow/{environment}/Scheduler',
                encryption_key=key,
                removal_policy=core.RemovalPolicy.DESTROY,
                retention=aws_logs.RetentionDays.INFINITE
            )

        logging_configuration = aws_mwaa.CfnEnvironment.LoggingConfigurationProperty(
            scheduler_logs=aws_mwaa.CfnEnvironment.ModuleLoggingConfigurationProperty(
                enabled=True,
                cloud_watch_log_group_arn=scheduler_log.log_group_arn,
                log_level='WARNING'
           )
        )

        mwaa_env = aws_mwaa.CfnEnvironment(
            scope=self,
            id=f'airflow-{environment}',
            name=f'{environment}',
            airflow_version='1.10.12',
            dag_s3_path='dags',
            environment_class='mw1.small',
            execution_role_arn=role.role_arn,
            kms_key=key.key_arn,
            logging_configuration=logging_configuration,
            max_workers=10,
            network_configuration=network_configuration,
            requirements_s3_path='requirements.txt',
            source_bucket_arn=bucket.bucket_arn,
            tags=None,
            webserver_access_mode='PRIVATE_ONLY'
        )

What did you expect to happen?

Successful deployment of the stack

What actually happened?

The following error occurred while calling cdk deploy

1/4 | 2:35:29 AM | UPDATE_IN_PROGRESS   | AWS::Logs::LogGroup         | airflow-cloudwatch-development-Scheduler (airflowcloudwatchdevelopmentScheduler657AD12B) Resource creation Initiated
1/4 | 2:35:29 AM | UPDATE_COMPLETE      | AWS::Logs::LogGroup         | airflow-cloudwatch-development-Scheduler (airflowcloudwatchdevelopmentScheduler657AD12B)
1/4 | 2:35:35 AM | UPDATE_FAILED        | AWS::MWAA::Environment      | airflow-development-det (airflowdevelopmentdet) Properties validation failed for resource airflowdevelopment with message:
#/LoggingConfiguration/SchedulerLogs/CloudWatchLogGroupArn: failed validation constraint for keyword [pattern]
        tmp\tmp_s9pvqvk\lib\program.js:8154:58
        \_ Kernel._wrapSandboxCode (tmp\tmp_s9pvqvk\lib\program.js:8582:24)
        \_ Kernel._create (tmp\tmp_s9pvqvk\lib\program.js:8154:34)
        \_ Kernel.create (tmp\tmp_s9pvqvk\lib\program.js:7895:29)
        \_ KernelHost.processRequest (tmp\tmp_s9pvqvk\lib\program.js:9479:36)
        \_ KernelHost.run (tmp\tmp_s9pvqvk\lib\program.js:9442:22)
        \_ Immediate._onImmediate (tmp\tmp_s9pvqvk\lib\program.js:9443:46)
        \_ processImmediate (internal/timers.js:456:21)

Environment

  • CDK CLI Version : 1.100.0 (build d996c6d)
  • Framework Version:
  • Node.js Version: v12.16.1
  • OS : Windows Server 2016 Datacenter
  • Language (Version): Python (3.8.6)

Other


This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
kostiantyn-privcommented, Mar 24, 2022

@Patrick-Postnl

My solution is

  1. Create LogGroups
       log_groups = {}
        for log_group in ['DAGProcessing', 'Scheduler', 'Task', 'WebServer', 'Worker']:
            log_groups[log_group] = aws_logs.LogGroup(
                scope=self,
                id=f'{environment}-{log_group}',
                log_group_name=f'airflow-{environment}-{log_group}',
                removal_policy=core.RemovalPolicy.DESTROY,
                retention=aws_logs.RetentionDays.SIX_MONTHS
            )
  1. Create LoggingConfigurationProperty
        logging_configuration = aws_mwaa.CfnEnvironment.LoggingConfigurationProperty(
            dag_processing_logs=aws_mwaa.CfnEnvironment.ModuleLoggingConfigurationProperty(
                enabled=True,
                cloud_watch_log_group_arn=log_groups['DAGProcessing'].log_group_arn,
                log_level='INFO'
            ),
            scheduler_logs=aws_mwaa.CfnEnvironment.ModuleLoggingConfigurationProperty(
                enabled=True,
                cloud_watch_log_group_arn=log_groups['Scheduler'].log_group_arn,
                log_level='INFO'
            ),
            task_logs=aws_mwaa.CfnEnvironment.ModuleLoggingConfigurationProperty(
                enabled=True,
                cloud_watch_log_group_arn=log_groups['Task'].log_group_arn,
                log_level='INFO'
            ),
            webserver_logs=aws_mwaa.CfnEnvironment.ModuleLoggingConfigurationProperty(
                enabled=False,
                cloud_watch_log_group_arn=log_groups['WebServer'].log_group_arn,
                log_level='INFO'
            ),
            worker_logs=aws_mwaa.CfnEnvironment.ModuleLoggingConfigurationProperty(
                enabled=True,
                cloud_watch_log_group_arn=log_groups['Worker'].log_group_arn,
                log_level='WARNING'
            )
        )
  1. And use it in mwaa
        mwaa_env = aws_mwaa.CfnEnvironment(
            scope=self,
            id='airflow',
            name=f'{environment}',
            airflow_configuration_options=None,
            airflow_version='1.10.12',
            dag_s3_path='dags',
            environment_class='mw1.small',
            execution_role_arn=role.role_arn,
            logging_configuration=logging_configuration,
            max_workers=10,
            network_configuration=network_configuration,
            plugins_s3_object_version=None,
            plugins_s3_path=None,
            requirements_s3_object_version=None,
            requirements_s3_path='requirements.txt',
            source_bucket_arn=bucket.bucket_arn,
            tags=None,
            webserver_access_mode='PRIVATE_ONLY',
            weekly_maintenance_window_start='SUN:03:30'
        )
0reactions
Patrick-Postnlcommented, Mar 24, 2022

@kostiantyn-priv what was your solution for this issue?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting: CloudWatch Logs and CloudTrail errors
Use the following topics to resolve issues related to Amazon CloudWatch Logs and AWS CloudTrail for an Amazon Managed Workflows for Apache Airflow...
Read more >
Deploying MWAA Using AWS CDK - DZone
Learn how to use a Python AWS CDK application to configure and deploy your Apache Airflow environments using MWAA in a repeatable and ......
Read more >
Using AWS CDK to deploy your Amazon ... - Beachgeek blog
If you deploy and are using an existing Amazon S3 bucket as your MWAA Dag bucket, the deployment will fail. Make sure that...
Read more >
Setting up MWAA to use a KMS key - DEV Community ‍ ‍
In a previous post, I shared how you can using AWS CDK to provision your Apache Airflow environments using the Managed Workflows for...
Read more >
Using AWS CDK to deploy your Amazon Managed Workflows ...
If you deploy and are using an existing Amazon S3 bucket as your MWAA Dag bucket, the deployment will fail. Make sure that...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found