list_blob_names is slow on my storage, returns markers
See original GitHub issueWhich service(blob, file, queue) does this issue concern?
blob
Which version of the SDK was used? Please provide the output of pip freeze
.
azure-storage-blob==2.1.0 azure-storage-common==2.1.0 azure-storage-file==2.1.0
What problem was encountered?
list_blob_names is extremely slow for a particular blob as it seems to be listing “markers” , never seen that before, I just call list_blob_names without num_results.
The response object is an empty list and a marker:
x-ms-date:Thu, 28 May 2020 15:43:56 GMT x-ms-version:2019-02-02 /alsordon/phillytools comp:list delimiter:/ marker:2!472!MDAwMzA5IXByb2plY3RzL2luZm9uY2VzL2MxMF9yZXMxOC9wdC1yZXN1bHRzL2FwcGxpY2F0aW9uXzE1ODMzMDcxNTM4NjhfMzIzNjMvU3VwQ29uL2NpZmFyMTBfdGVuc29yYm9hcmQvSW5mb05DRXNfY2lmYXIxMF9yZXNuZXQxOF9scl8wLjVfZGVjYXlfMC4wMDAxX2Jzel8zMDcyX3RlbXBfMC41X2wxMl8xLjAwX2wxY3V0Ml8xLjAwX2wxMkkxY3V0XzEuMDBfbnNfNTEyX3Zhcl8wLjEwMF90cmlhbF8wX2Nvc2luZV93YXJtL2V2ZW50cy5vdXQudGZldmVudHMuMTU5MDM0ODMxOS5jb250YWluZXItZTc3OC0xNTgzMzA3MTUzODY4LTMyMzYzLTAxLTAwMDAwMiEwMDAwMjghMjAyMC0wNS0yNVQwMzo0Mzo0OS41NjEyMDg5WiE- prefix:projects/infonces/ restype:container 2020-05-28 11:44:01 AM - DD https://alsordon.blob.core.windows.net:443 “GET /phillytools?restype=container&comp=list&prefix=projects%2Finfonces%2F&delimiter=%2F&marker=2%21472%21MDAwMzA5IXByb2plY3RzL2luZm9uY2VzL2MxMF9yZXMxOC9wdC1yZXN1bHRzL2FwcGxpY2F0aW9uXzE1ODMzMDcxNTM4NjhfMzIzNjMvU3VwQ29uL2NpZmFyMTBfdGVuc29yYm9hcmQvSW5mb05DRXNfY2lmYXIxMF9yZXNuZXQxOF9scl8wLjVfZGVjYXlfMC4wMDAxX2Jzel8zMDcyX3RlbXBfMC41X2wxMl8xLjAwX2wxY3V0Ml8xLjAwX2wxMkkxY3V0XzEuMDBfbnNfNTEyX3Zhcl8wLjEwMF90cmlhbF8wX2Nvc2luZV93YXJtL2V2ZW50cy5vdXQudGZldmVudHMuMTU5MDM0ODMxOS5jb250YWluZXItZTc3OC0xNTgzMzA3MTUzODY4LTMyMzYzLTAxLTAwMDAwMiEwMDAwMjghMjAyMC0wNS0yNVQwMzo0Mzo0OS41NjEyMDg5WiE- HTTP/1.1” 200 None 2020-05-28 11:44:01 AM - DD String_to_sign=GET
x-ms-client-request-id:0da62ca2-a0fa-11ea-a479-554857c9fd70 x-ms-date:Thu, 28 May 2020 15:44:01 GMT x-ms-version:2019-02-02 /alsordon/phillytools comp:list delimiter:/ marker:2!400!MDAwMjU1IXByb2plY3RzL2luZm9uY2VzL2MxMF9yZXMxOC9wdC1yZXN1bHRzL2FwcGxpY2F0aW9uXzE1ODMzMDcxNTM4NjhfMzIzNjYvU3VwQ29uL2NpZmFyMTBfdGVuc29yYm9hcmQvU2ltQ0xSX2NpZmFyMTBfcmVzbmV0MThfbHJfMC41X2RlY2F5XzAuMDAwMV9ic3pfMzA3Ml90ZW1wXzAuNV90cmlhbF8wX2Nvc2luZV93YXJtL2V2ZW50cy5vdXQudGZldmVudHMuMTU5MDM0ODQyMC5jb250YWluZXItZTc3OC0xNTgzMzA3MTUzODY4LTMyMzY2LTAxLTAwMDAwMiEwMDAwMjghMjAyMC0wNS0yNFQyMjowNjo1NS40NzUzMDk4WiE- prefix:projects/infonces/ restype:container
Have you found a mitigation/solution?
No
Note: for table service, please post the issue here instead: https://github.com/Azure/azure-cosmosdb-python.
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (3 by maintainers)
Hi @sordonia
Thanks for reaching out! Are you doing something like this next(self.bs.list_blobs(container_name))? Just want to clarify that the “log” you pasted there are requests. If there’s a marker in request that means the response returned that marker, then the response should have returned the list of blobs.
BTW you said “never seen that before” so does that mean this was working and suddenly it’s not working? if that’s the case we will look into service log and see what happens!
Thanks!
Hi @erlisb azure-storage has been deprecated, would you like to use azure-storage-blob<=2.1.0 or azure-storage-blob>=12.0.0 (there are be breaking changes if you use >=12.0.0)
@zezha-msft do you have any idea about the list_blobs behavior in azure-storage==0.36…