LocalFilesystemToS3Operator dest_key can not be a full s3:// style url

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.3.0 apache-airflow-providers-ftp==2.1.2 apache-airflow-providers-http==2.1.2 apache-airflow-providers-imap==2.2.3 apache-airflow-providers-mongo==2.3.3 apache-airflow-providers-sqlite==2.1.3

Apache Airflow version

2.3.0b1 (pre-release)

Operating System

Arch Linux

Deployment

Virtualenv installation

Deployment details

No response

What happened

LocalFilesystemToS3Operator does not accept full s3:// style url as dest_key, although it states, that it should:

    :param dest_key: The key of the object to copy to. (templated)

        It can be either full s3:// style url or relative path from root level.

        When it's specified as a full s3:// url, including dest_bucket results in a TypeError.

What you think should happen instead

LocalFilesystemToS3Operator should behave as documented.

How to reproduce

A modification of an existing UT:

    @mock_s3
    def test_execute_with_only_key(self):
        conn = boto3.client('s3')
        conn.create_bucket(Bucket=self.dest_bucket)
        operator = LocalFilesystemToS3Operator(
            task_id='s3_to_file_sensor',
            dag=self.dag,
            filename=self.testfile1,
            dest_key=f's3://dummy/{self.dest_key}',
            **self._config,
        )
        operator.execute(None)

        objects_in_dest_bucket = conn.list_objects(Bucket=self.dest_bucket, Prefix=self.dest_key)
        # there should be object found, and there should only be one object found
        assert len(objects_in_dest_bucket['Contents']) == 1
        # the object found should be consistent with dest_key specified earlier
        assert objects_in_dest_bucket['Contents'][0]['Key'] == self.dest_key

FAILED tests/providers/amazon/aws/transfers/test_local_to_s3.py::TestFileToS3Operator::test_execute_with_only_key - TypeError: expected string or bytes-like object

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[vincbeck]

After investigation, it seems it is not the only S3 related operator with this bug. Thanks for catching this! Unless you want or are currently working on this, I can take care of fixing these operators

[potiuk]

Would you like to fix it @tanelk ? Sounds like an easy fix and you can become one of > 2000 contriibutors to airflow. Actually fixing it is the fastest way to proceed, otherwise it will have to wait for someone willing to pick it up.