Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

list_blob_names is slow on my storage, returns markers

See original GitHub issue

Which service(blob, file, queue) does this issue concern?


Which version of the SDK was used? Please provide the output of pip freeze.

azure-storage-blob==2.1.0 azure-storage-common==2.1.0 azure-storage-file==2.1.0

What problem was encountered?

list_blob_names is extremely slow for a particular blob as it seems to be listing “markers” , never seen that before, I just call list_blob_names without num_results.

The response object is an empty list and a marker:

x-ms-date:Thu, 28 May 2020 15:43:56 GMT x-ms-version:2019-02-02 /alsordon/phillytools comp:list delimiter:/ marker:2!472!MDAwMzA5IXByb2plY3RzL2luZm9uY2VzL2MxMF9yZXMxOC9wdC1yZXN1bHRzL2FwcGxpY2F0aW9uXzE1ODMzMDcxNTM4NjhfMzIzNjMvU3VwQ29uL2NpZmFyMTBfdGVuc29yYm9hcmQvSW5mb05DRXNfY2lmYXIxMF9yZXNuZXQxOF9scl8wLjVfZGVjYXlfMC4wMDAxX2Jzel8zMDcyX3RlbXBfMC41X2wxMl8xLjAwX2wxY3V0Ml8xLjAwX2wxMkkxY3V0XzEuMDBfbnNfNTEyX3Zhcl8wLjEwMF90cmlhbF8wX2Nvc2luZV93YXJtL2V2ZW50cy5vdXQudGZldmVudHMuMTU5MDM0ODMxOS5jb250YWluZXItZTc3OC0xNTgzMzA3MTUzODY4LTMyMzYzLTAxLTAwMDAwMiEwMDAwMjghMjAyMC0wNS0yNVQwMzo0Mzo0OS41NjEyMDg5WiE- prefix:projects/infonces/ restype:container 2020-05-28 11:44:01 AM - DD “GET /phillytools?restype=container&comp=list&prefix=projects%2Finfonces%2F&delimiter=%2F&marker=2%21472%21MDAwMzA5IXByb2plY3RzL2luZm9uY2VzL2MxMF9yZXMxOC9wdC1yZXN1bHRzL2FwcGxpY2F0aW9uXzE1ODMzMDcxNTM4NjhfMzIzNjMvU3VwQ29uL2NpZmFyMTBfdGVuc29yYm9hcmQvSW5mb05DRXNfY2lmYXIxMF9yZXNuZXQxOF9scl8wLjVfZGVjYXlfMC4wMDAxX2Jzel8zMDcyX3RlbXBfMC41X2wxMl8xLjAwX2wxY3V0Ml8xLjAwX2wxMkkxY3V0XzEuMDBfbnNfNTEyX3Zhcl8wLjEwMF90cmlhbF8wX2Nvc2luZV93YXJtL2V2ZW50cy5vdXQudGZldmVudHMuMTU5MDM0ODMxOS5jb250YWluZXItZTc3OC0xNTgzMzA3MTUzODY4LTMyMzYzLTAxLTAwMDAwMiEwMDAwMjghMjAyMC0wNS0yNVQwMzo0Mzo0OS41NjEyMDg5WiE- HTTP/1.1” 200 None 2020-05-28 11:44:01 AM - DD String_to_sign=GET

x-ms-client-request-id:0da62ca2-a0fa-11ea-a479-554857c9fd70 x-ms-date:Thu, 28 May 2020 15:44:01 GMT x-ms-version:2019-02-02 /alsordon/phillytools comp:list delimiter:/ marker:2!400!MDAwMjU1IXByb2plY3RzL2luZm9uY2VzL2MxMF9yZXMxOC9wdC1yZXN1bHRzL2FwcGxpY2F0aW9uXzE1ODMzMDcxNTM4NjhfMzIzNjYvU3VwQ29uL2NpZmFyMTBfdGVuc29yYm9hcmQvU2ltQ0xSX2NpZmFyMTBfcmVzbmV0MThfbHJfMC41X2RlY2F5XzAuMDAwMV9ic3pfMzA3Ml90ZW1wXzAuNV90cmlhbF8wX2Nvc2luZV93YXJtL2V2ZW50cy5vdXQudGZldmVudHMuMTU5MDM0ODQyMC5jb250YWluZXItZTc3OC0xNTgzMzA3MTUzODY4LTMyMzY2LTAxLTAwMDAwMiEwMDAwMjghMjAyMC0wNS0yNFQyMjowNjo1NS40NzUzMDk4WiE- prefix:projects/infonces/ restype:container

Have you found a mitigation/solution?


Note: for table service, please post the issue here instead:

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

xiafu-msftcommented, May 31, 2020

Hi @sordonia

Thanks for reaching out! Are you doing something like this next( Just want to clarify that the “log” you pasted there are requests. If there’s a marker in request that means the response returned that marker, then the response should have returned the list of blobs.

BTW you said “never seen that before” so does that mean this was working and suddenly it’s not working? if that’s the case we will look into service log and see what happens!


xiafu-msftcommented, Jun 25, 2020

Hi @erlisb azure-storage has been deprecated, would you like to use azure-storage-blob<=2.1.0 or azure-storage-blob>=12.0.0 (there are be breaking changes if you use >=12.0.0)

@zezha-msft do you have any idea about the list_blobs behavior in azure-storage==0.36…

Read more comments on GitHub >

github_iconTop Results From Across the Web

Working with delete markers - Amazon Simple Storage Service
Deleting a versioned object without a version ID creates a placeholder or marker that makes the object appear as though it is deleted...
Read more >
Android Google Map addMarker() very slow when adding 400 ...
It is my understanding that each time you draw a marker it sends the drawing operations to a thread pool in the background...
Read more >
Quantification of protein markers monitoring the pre-analytical ...
In the hospital, blood samples are collected to monitor patients' health states ... of protein markers monitoring the pre-analytical effect of blood storage...
Read more >
How do I mass delete 'Delete-Markers' inside a bucket?
Sometimes due to the S3 versioning feature, a bucket can accumulate a large ... part of LIST operation calls to any S3 storage...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found