question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

If the key of the object contains control characters, delete () of s3.Bucket.objectsCollection fails.

See original GitHub issue

Is there a way to delete these objects while avoiding any errors while using batch delete?

reproduce

  • upload new object which include /\x10 in file name.
  • try batch delete with s3.Bucket.objectsCollection
objs = bucket.objects.filter(Prefix="path/to/includes_escape")
for obj in objs.all():
    print(repr((obj.key)))

-->
'path/to/includes_escape/\x10\x10\x10\x10\x10\x10\x10foo.jpg'
'path/to/includes_escape/\x10\x10\x10\x10\x10\x10\x10bar.jpg'
objs.delete()

--> ClientError: An error occurred (MalformedXML) when calling the DeleteObjects operation: The XML you provided was not well-formed or did not validate against our published schema

environment

  • aws lambda python 3.7
  • boto3==1.9.42

(edit)

debug trace

boto3.resources.collection [DEBUG] Calling paginated s3:list_objects with {'Bucket': 'sawadev-s3object-filter', 'Prefix': 'path/to/includes_escape'}
boto3.resources.factory [DEBUG] Loading s3:ObjectSummary
boto3.resources.model [DEBUG] Renaming ObjectSummary attribute key
boto3.resources.action [DEBUG] Calling s3:delete_objects with {
  'Bucket': 'my-example-bucket',
  'Delete': {'Objects': [
    {'Key': 'path/to/includes_escape/\x10\x10\x10\x10\x10\x10\x10foo.jpg'},
    {'Key': 'path/to/includes_escape/\x10\x10\x10\x10\x10\x10\x10bar.jpg'}
  ]}}

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:9 (3 by maintainers)

github_iconTop GitHub Comments

5reactions
bardiakhosravicommented, Oct 14, 2019

I am still seeing this issue. Has there been any resolution to this?

0reactions
no-response[bot]commented, Jul 25, 2019

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don’t have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Cannot delete Amazon S3 key that contains bad character
This solved the issue, just delete the main folder: aws s3 rm "s3://BUCKET_NAME/folder/folder" --recursive.
Read more >
Collections — Boto3 Docs 1.26.32 documentation - AWS
Overview¶. A collection provides an iterable interface to a group of resources. Collections behave similarly to Django QuerySets and expose a ...
Read more >
Read file content from S3 bucket with boto3 - Edureka
I read the filenames in my S3 bucket by doing. objs = boto3.client.list_objects(Bucket='my_bucket') while 'Contents' in objs.keys(): ...
Read more >
Listing object keys programmatically - AWS Documentation
Amazon S3 exposes a list operation that lets you enumerate the keys contained in a bucket. Keys are selected for listing by bucket...
Read more >
Collect Logs from Your OCI Object Storage Bucket
When you configure a bucket for log collection, Oracle Logging ... only delete the associated configuration with the bucket but will have no ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found