question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Loads of DEBUG messages with new s3transfer package

See original GitHub issue

I just updated my boto3 and it installed a new s3transfer package (0.1.1). Problem is that now I get a load of DEBUG messages for every S3 download my app performs.

DEBUG:s3transfer.utils: Acquiring 0
DEBUG:s3transfer.tasks: DownloadSubmissionTask({'transfer_future': <s3transfer.futures.TransferFuture object at 0x10ae6a350>}) about to wait for the following futures []
DEBUG:s3transfer.tasks: DownloadSubmissionTask({'transfer_future': <s3transfer.futures.TransferFuture object at 0x10ae6a350>}) done waiting for dependent futures
DEBUG:s3transfer.tasks: Executing task DownloadSubmissionTask({'transfer_future': <s3transfer.futures.TransferFuture object at 0x10ae6a350>}) with kwargs {'io_executor': <s3transfer.futures.BoundedExecutor object at 0x10ae34f90>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x10ae34b90>, 'osutil': <s3transfer.utils.OSUtils object at 0x10ae34950>, 'client': <botocore.client.S3 object at 0x10ae343d0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x10ae6a350>, 'config': <boto3.s3.transfer.TransferConfig object at 0x10ae34910>}
DEBUG:s3transfer.futures: Submitting task GetObjectTask({'extra_args': {}, 'bucket': 'BUCKET', 'key': 'FILE'}) to executor <s3transfer.futures.BoundedExecutor object at 0x10ae34b90> for transfer request: 0.
DEBUG:s3transfer.utils: Acquiring 0
DEBUG:s3transfer.tasks: GetObjectTask({'extra_args': {}, 'bucket': 'BUCKET', 'key': 'FILE'}) about to wait for the following futures []
DEBUG:s3transfer.tasks: GetObjectTask({'extra_args': {}, 'bucket': 'BUCKET', 'key': 'FILE'}) done waiting for dependent futures
DEBUG:s3transfer.tasks: Executing task GetObjectTask({'extra_args': {}, 'bucket': 'BUCKET', 'key': 'FILE'}) with kwargs {'fileobj': <open file '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f.6A5DbD3C', mode 'wb' at 0x109865a50>, 'bucket': 'BUCKET', 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x10ae90890>, 'extra_args': {}, 'callbacks': [], 'client': <botocore.client.S3 object at 0x10ae343d0>, 'key': 'FILE', 'io_chunksize': 262144, 'max_attempts': 5}
DEBUG:s3transfer.futures: Submitting task JoinFuturesTask({}) to executor <s3transfer.futures.BoundedExecutor object at 0x10ae34b90> for transfer request: 0.
DEBUG:s3transfer.utils: Acquiring 0
DEBUG:s3transfer.tasks: JoinFuturesTask({}) about to wait for the following futures [<Future at 0x10aeb9390 state=running>]
DEBUG:s3transfer.utils: Releasing acquire 0/None
DEBUG:s3transfer.tasks: JoinFuturesTask({}) about to wait for <Future at 0x10aeb9390 state=running>
DEBUG:s3transfer.futures: Submitting task IOWriteTask({'offset': 0}) to executor <s3transfer.futures.BoundedExecutor object at 0x10ae34f90> for transfer request: 0.
DEBUG:s3transfer.utils: Acquiring 0
DEBUG:s3transfer.tasks: IOWriteTask({'offset': 0}) about to wait for the following futures []
DEBUG:s3transfer.tasks: IOWriteTask({'offset': 0}) done waiting for dependent futures
DEBUG:s3transfer.tasks: Executing task IOWriteTask({'offset': 0}) with kwargs {'fileobj': <open file '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f.6A5DbD3C', mode 'wb' at 0x109865a50>, 'offset': 0}
DEBUG:s3transfer.utils: Releasing acquire 0/None
DEBUG:s3transfer.tasks: JoinFuturesTask({}) done waiting for dependent futures
DEBUG:s3transfer.utils: Releasing acquire 0/None
DEBUG:s3transfer.tasks: Executing task JoinFuturesTask({}) with kwargs {'futures_to_wait_on': [None]}
DEBUG:s3transfer.futures: Submitting task IORenameFileTask({'final_filename': '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f'}) to executor <s3transfer.futures.BoundedExecutor object at 0x10ae34f90> for transfer request: 0.
DEBUG:s3transfer.utils: Acquiring 0
DEBUG:s3transfer.utils: Releasing acquire 0/None
DEBUG:s3transfer.tasks: IORenameFileTask({'final_filename': '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f'}) about to wait for the following futures []
DEBUG:s3transfer.tasks: IORenameFileTask({'final_filename': '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f'}) done waiting for dependent futures
DEBUG:s3transfer.tasks: Executing task IORenameFileTask({'final_filename': '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f'}) with kwargs {'osutil': <s3transfer.utils.OSUtils object at 0x10ae34950>, 'final_filename': '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f', 'fileobj': <open file '/var/folders/kd/yx5n34z97kz10x25_5z9sq9m9qvmpm/T/tmpgrD02f.6A5DbD3C', mode 'wb' at 0x109865a50>}
DEBUG:s3transfer.utils: Releasing acquire 0/None

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Reactions:4
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

4reactions
pvlbzncommented, Apr 1, 2018

Set s3transfer to CRITICAL only.

logging.getLogger('s3transfer').setLevel(logging.CRITICAL)
2reactions
kyleknapcommented, Aug 9, 2016

Ahh I see what is happening. We need to add a null handler default in s3transfer. I sent a PR to fix this in the s3transfer library.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Disable boto logging without modifying the boto files
It is recommended to disable DEBUG and INFO messages from all external packages, not limited to botocore and boto3 : LOGGING_CONFIG = {...
Read more >
Considerations in adopting RHEL 8 Red Hat Enterprise Linux 8
New syslog plug-in for logging any RPM activity by the System Logging protocol (syslog). The rpmbuild command can now do all build steps...
Read more >
s3transfer - PyPI
S3transfer is a Python library for managing Amazon S3 transfers. This project is maintained and published by Amazon Web Services.
Read more >
Using credentials - AWS SDK for Java 2.x
The SDK uses the SystemPropertyCredentialsProvider class to load credentials from the aws.accessKeyId and aws.secretAccessKey Java system properties.
Read more >
8 Must-Know Tricks to Use S3 More Effectively in Python
Similarly, if you want to upload and read small pieces of textual data such as quotes, tweets, or news articles, you can do...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found