question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Uploading 'empty' blocks to page blobs is wasteful.

See original GitHub issue

Re: PageBlobService#create_blob_from_stream

When a page blob is provisioned, the content is a stream of 0s. Therefore, when uploading a page blob, any block that is all zeros does not need to be uploaded, it can simply be skipped.

Most fixed-length VHD images are mostly empty blocks, so uploading all those zeros represents a huge waste of time and bandwidth.

Our optimum solution for dealing with this in azurectl was to load a block from /dev/zero and compare it to the current block in the stream. If they are equal… move along to the next block without uploading.

See https://github.com/SUSE/azurectl/blob/master/azurectl/storage/page_blob.py#L50 for example.

And, just for reference, a 30GB openSUSE Leap 42.3 image, contains < 300MB of full blocks. Uploading with the API convenience methods, or az, results in an upload of 29.7GB of zeros. o.0

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Reactions:1
  • Comments:7 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
evanrolfecommented, Jan 11, 2018
0reactions
zezha-msftcommented, Jan 23, 2018

This was fixed in release 1.0.0 of the blob package.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Azure blob's block list is empty, but blob is not empty! How can ...
This issue in a nutshell: A block blob can be created with a single PUT request. This will create a blob with committed...
Read more >
Uploading blobs to Microsoft Azure - The robust way
We're talking 'Block' Blobs​​ Azure blob storage supports two different types of blobs, page blobs and block blobs: Usually, a page blob is ......
Read more >
Do's and Don'ts for Streaming File Uploads to Azure Blob ...
This is slow and it is wasteful if all we want to do is forward the data right on to Azure Blob Storage....
Read more >
Put Blob (REST API) - Azure Storage - Microsoft Learn
The Put Blob operation creates a new block, page, or append blob, or updates the content of an existing block blob.
Read more >
veeam backup to azure blob BAD performance
The MS documentation is very clear that while there is a 60 MB/s throughput limit on writing of single page blob, the block...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found