Error on logging empty line to Cloudwatch

Apache Airflow version: 2.0.1

Environment:

• Cloud provider or hardware configuration: AWS

What happened: I have Airflow with Cloudwatch-based remote logging running. I also have BashOperator that does, for example, rsync with invalid parameters, for example rsync -av test test. The output of the rsync error is formatted and contains empty line. Once that empty line is logged to the Cloudwatch, i receive an error:

2021-04-06 19:29:22,318] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:154 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[5].message, value: 0, valid range: 1-inf
[2021-04-06 19:29:22,320] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:158 WatchtowerWarning: Failed to deliver logs: None

So basically empty lines can’t be submitted to the Cloudwatch and as result the whole output of the process doesn’t appear in logs.

What you expected to happen:

I expect to have an output of the bash command in logs. Empty lines can be skipped or replaced with something.

How to reproduce it:

For example: run BashOperator with rsync command that fails on Airflow with Cloudwatch-based remote logging. It could be any other command that produces empty line in the output.

[o-nikolas]

I cut an issue to the watchtower repo (since the other was closed) with some possible fixes I had tested. But it seems like there is already a fix that just got released into Watchtower 2.0.

I have yet to do some testing with the new release, but once I get around to (and assuming it works as expected) it should be an easy fix to bump the watchtower version in Airflow to 2.0.

I’m happy to be assigned this issue if no one else is working on it.

[sicarul]

I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i’ll test it later 😄