rsync with s3 bucket BadRequestException: 400 AuthorizationHeaderMalformed

Good morning all,

I used to rsync data between one of my buckets in GCS in an S3 one. This morning I tried to run the same command as always but doesn’t work anymore.

This is the error I receiving now: Caught non-retryable exception while listing s3://M-S3-BUCKET/: BadRequestException: 400 AuthorizationHeaderMalformed And this one: CommandException: Caught non-retryable exception - aborting rsync

This task for us it’s really important, how can I fix it?

thanks so much for your time.

With all the best, Fabio Rigato

[dilipped]

This is happening because we recently started using the V4 signature for S3. The new signature depends on the region information as well. Here your bucket is not present in the default region us-east-1 but it is in the ap-southeast-2 region. Since the region was not specified, gsutil tries with the default region to calculate the signature and fails. We will soon be adding a retry logic to fix this. Meanwhile, you can try the temp fix given below:

Region information is derived from the host, and to fix this, you can add the host information in the .boto file under s3 section

[s3]
host = s3.ap-southeast-2.amazonaws.com

or you can pass this information from the top-level options gsutil -o "s3:host=s3.ap-southeast-2.amazonaws.com" ...

Note that this will use region as ap-southeast-2 for the entire execution of the command ,so if you have buckets in multiple regions, then you might want to run separate commands for each region.

[dilipped]

You’re welcome. I will keep this issue open to track the retry logic fix that I have mentioned above.