question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Writing to S3 with server-side encryption doesn't work anymore

See original GitHub issue

Problem description

Earlier versions of smart_open (I’ve tested v1.11.1 and v3.0.0) allowed writing to S3 with server-side encryption like this:

output_handle = smart_open.open(out_url, "w", transport_params={         'multipart_upload_kwargs': {             'ServerSideEncryption': 'AES256'         }})

In the latest versions this doesn’t work and I cannot find any documentation or figure out a different way of making it work. The result is I get an error like ValueError: the bucket 'myBucket' does not exist, or is forbidden for access (ClientError('An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied'))

Steps/code to reproduce the problem

  • install the latest version of smart_open
  • create an output handle as shown above

Versions

Please provide the output of:

import platform, sys, smart_open
print(platform.platform())
print("Python", sys.version)
print("smart_open", smart_open.__version__)

Linux-4.4.0-17763-Microsoft-x86_64-with-glibc2.10 Python 3.8.2 (default, Mar 26 2020, 15:53:00) [GCC 7.3.0] smart_open 5.0.0

Checklist

Before you create the issue, please make sure you have:

  • Described the problem clearly
  • Provided a minimal reproducible example, including any required data
  • Provided the version numbers of the relevant software

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9

github_iconTop GitHub Comments

2reactions
mpenkovcommented, May 7, 2021

Try this:

client_kwargs = {
  'S3.Client.create_multipart_upload': {
    'ServerSideEncryption': 'AES256',
  }
}
writer = smart_open.open(url, 'w', transport_params={'client_kwargs': client_kwargs})
1reaction
challisdcommented, Aug 17, 2021

Sadly no, my old WSL environment got deleted with the last Windows 10 upgrade (thanks Microsoft!)

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot data delivery failure between Kinesis and S3
Kinesis Data Firehose supports Amazon S3 server-side encryption with AWS Key Management Service (AWS KMS) for encrypting data that's ...
Read more >
Issues while reading and writing a KMS encrypted spark data ...
The bucket has server-side encryption setup. I'm having the following packages run from spark-default.conf: spark.jars.packages com.amazonaws: ...
Read more >
S3 KMS encryption support · Issue #348 · shrinerb/shrine
We are using the KMS encryption support to upload/download files from S3. I managed to make it work by overwriting the S3 client...
Read more >
Python, Boto3, and AWS S3: Demystified - Real Python
Get started working with Python, Boto3, and AWS S3. Learn how to create objects, ... Create a new file and upload it using...
Read more >
All you need to know about encrypting AWS S3 buckets
Dance like nobody's watching, encrypt like everyone is. Learn how to enable S3 default encryption. But that's not enough.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found