question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Failing to break lease for PREMIUM_LRS storage or detach os disk

See original GitHub issue

I have several virtual machines which uses a bunch of blobs. these blobs are divided into two storages (PREMIUM_LRS and STANDARD_LRS). If i try to restore a blob, i shutdown the virutel machines which is using the blob and then break the lease (instead of deleting the VM). this works fine for STANDARD_LRS stored blobs. For PREMIUM_LRS Blobs i allways get the Message: “This blob is being used by the system.” - but i did stop and deallocated the using virtuel machine?!

DEBUG LOG:

File logging enabled - Writing logs to '/root/.azure/logs'.
Command arguments ['storage', 'blob', 'lease', 'break', '-b', 'test02-db01-02.vhd', '-c', 'vhds', '--connection-string', 'DefaultEndpointsProtocol=https;EndpointSuffix=core.cloudapi.de;AccountName=[REPLACED];AccountKey=[REPLACED]']
Current cloud config:
{'endpoints': {'active_directory': 'https://login.microsoftonline.de',
               'active_directory_data_lake_resource_id': None,
               'active_directory_graph_resource_id': 'https://graph.cloudapi.de/',
               'active_directory_resource_id': 'https://management.core.cloudapi.de/',
               'batch_resource_id': 'https://batch.cloudapi.de/',
               'gallery': 'https://gallery.cloudapi.de/',
               'management': 'https://management.core.cloudapi.de/',
               'resource_manager': 'https://management.microsoftazure.de',
               'sql_management': 'https://management.core.cloudapi.de:8443/',
               'vm_image_alias_doc': 'https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/arm-compute/quickstart-templates/aliases.json'},
 'is_active': True,
 'name': 'AzureGermanCloud',
 'profile': 'latest',
 'suffixes': {'azure_datalake_analytics_catalog_and_job_endpoint': None,
              'azure_datalake_store_file_system_endpoint': None,
              'keyvault_dns': '.vault.microsoftazure.de',
              'sql_server_hostname': '.database.cloudapi.de',
              'storage_endpoint': 'core.cloudapi.de'}}
Registered application event handler 'CommandTableParams.Loaded' at <function add_id_parameters at 0x7f41b36039d8>
Registered application event handler 'CommandTable.Loaded' at <function add_id_parameters at 0x7f41b36039d8>
Successfully loaded command table from module 'storage'.
Extensions directory: '/root/.azure/cliextensions'
Application event 'CommandTable.Loaded' with event data {'command_table': {'storage blob lease break': <azure.cli.core.commands.CliCommand object at 0x7f41b35c5ef0>}}
Application event 'CommandParser.Loaded' with event data {'parser': AzCliCommandParser(prog='az', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)}
Application event 'CommandTableParams.Loaded' with event data {'command_table': {'storage blob lease break': <azure.cli.core.commands.CliCommand object at 0x7f41b35c5ef0>}}
Application event 'CommandParser.Parsing' with event data {'argv': ['storage', 'blob', 'lease', 'break', '-b', 'test02-db01-02.vhd', '-c', 'vhds', '--connection-string', 'DefaultEndpointsProtocol=https;EndpointSuffix=core.cloudapi.de;AccountName=[REPLACED];AccountKey=[REPLACED]']}
Application event 'CommandParser.Parsed' with event data {'command': 'storage blob lease break', 'args': Namespace(_command_package='storage', _jmespath_query=None, _log_verbosity_debug=False, _log_verbosity_verbose=False, _output_format='json', _parser=AzCliCommandParser(prog='az storage blob lease break', usage=None, description='Breaks the lease, if the blob has an active lease. Once a lease is broken, it cannot be renewed. Any authorized request can break the lease; the request is not required to specify a matching lease ID. When a lease is  [...]
Getting data service client service_type=BlockBlobService
azure.multiapi.storage.v2017_04_17.common._auth : String_to_sign=PUT


x-ms-client-request-id:fd4b8d8e-da4f-11e7-a7b7-a434d9460bfd
x-ms-date:Wed, 06 Dec 2017 06:37:55 GMT
x-ms-lease-action:break
x-ms-version:2017-04-17
/test02fast/vhds/test02-db01-02.vhd
comp:lease
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Outgoing request: Method=PUT, Path=/vhds/test02-db01-02.vhd, Query={'comp': 'lease', 'timeout': None}, Headers={'x-ms-lease-id': None, 'x-ms-lease-action': 'break', 'x-ms-lease-duration': None, 'x-ms-lease-break-period': None, 'x-ms-proposed-lease-id': None, 'If-Modified-Since': None, 'If-Unmodified-Since': None, 'If-Match': None, 'If-None-Match': None, 'x-ms-version': '2017-04-17', 'User-Agent': 'Azure-Storage/0.37.0-0.36.0 (Python CPython 3.6.1; Linux 4.4.0-43-Microsoft) AZURECLI/2.0.20', 'x-ms-client-request-id': 'fd4b8d8e-da4f-11e7-a7b7-a434d9460bfd', 'x-ms-date': 'Wed, 06 Dec 2017 06:37:55 GMT', 'Authorization': 'SharedKey [REPLACED]'}.
urllib3.connectionpool : Starting new HTTPS connection (1): test02fast.blob.core.cloudapi.de
urllib3.connectionpool : https://test02fast.blob.core.cloudapi.de:443 "PUT /vhds/test02-db01-02.vhd?comp=lease HTTP/1.1" 409 219
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Receiving Response: Server-Timestamp=Wed, 06 Dec 2017 06:37:55 GMT, Server-Request-ID=1f00da01-001c-0014-045c-6eb6ae000000, HTTP Status Code=409, Message=This blob is being used by the system., Headers={'x-ms-lease-id': None, 'x-ms-lease-action': 'break', 'x-ms-lease-duration': None, 'x-ms-lease-break-period': None, 'x-ms-proposed-lease-id': None, 'If-Modified-Since': None, 'If-Unmodified-Since': None, 'If-Match': None, 'If-None-Match': None, 'x-ms-version': '2017-04-17', 'User-Agent': 'Azure-Storage/0.37.0-0.36.0 (Python CPython 3.6.1; Linux 4.4.0-43-Microsoft) AZURECLI/2.0.20', 'x-ms-client-request-id': 'fd4b8d8e-da4f-11e7-a7b7-a434d9460bfd', 'x-ms-date': 'Wed, 06 Dec 2017 06:37:55 GMT', 'Authorization': 'SharedKey test02fast:[REPLACED]'}.
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Wed, 06 Dec 2017 06:37:55 GMT, Server-Request-ID=1f00da01-001c-0014-045c-6eb6ae000000, HTTP status code=409, Exception=This blob is being used by the system.<?xml version="1.0" encoding="utf-8"?><Error><Code>SystemInUse</Code><Message>This blob is being used by the system.RequestId:1f00da01-001c-0014-045c-6eb6ae000000Time:2017-12-06T06:37:56.3333175Z</Message></Error>.
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Retry policy did not allow for a retry: Server-Timestamp=Wed, 06 Dec 2017 06:37:55 GMT, Server-Request-ID=1f00da01-001c-0014-045c-6eb6ae000000, HTTP status code=409, Exception=This blob is being used by the system.<?xml version="1.0" encoding="utf-8"?><Error><Code>SystemInUse</Code><Message>This blob is being used by the system.RequestId:1f00da01-001c-0014-045c-6eb6ae000000Time:2017-12-06T06:37:56.3333175Z</Message></Error>.

This blob is being used by the system.

Environment summary

Install Method (e.g. pip, interactive script, apt-get, Docker, MSI, edge build) / CLI version (az --version) / OS version / Shell Type (e.g. bash, cmd.exe, Bash on Windows)

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Reactions:1
  • Comments:27 (12 by maintainers)

github_iconTop GitHub Comments

1reaction
erosen03commented, Mar 1, 2018

Thanks @williexu and @mirobers. I don’t want to get off topic, but two other common scenarios that also currently require recreating a VM are:

  1. Moving a VM to a different subnet or vnet
  2. Adding a VM to or removing a VM from an availability or scale set

As were’re looking at the ‘recreate VM’ requirement regarding an OS disk, it would be really nice if we come up with a solution that would also help the other two scenarios listed above.

Thanks.

1reaction
williexucommented, Mar 1, 2018

reopening since there seems to be further discussion

Read more comments on GitHub >

github_iconTop Results From Across the Web

Breaking lease Standard vs Premium Storage - MSDN
Hi - We do not recommend breaking the lease on disks as that would cause the running VMs to crash. That is why...
Read more >
Unable to break lease and expand Ubuntu OS disk on Azure
If I go to Disks blade under the VM in ARM, I can see the detach option is grayed out. How can I...
Read more >
VM Broken? Use OS disk swap in Azure to fix and restore
In summary, I create a copy of the disk, fix the issues and then swap the OS drive. You cannot detach the OS...
Read more >
Break lease on Azure Blob Storage
Failed to delete blob 'vhds/<nameofblob>.vhd'. Error: There is currently a lease on the blob and no lease ID was specified in the request....
Read more >
Other Important Laws - Landlords & Tenants
With proper written notice, a landlord can end a month-to-month tenancy unless ... The landlord must remove, store and take care of the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found