question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Encryption failing on Ubuntu 16.04

See original GitHub issue

I’m building Azure Disk Encryption into an ARM template and have configured encryption of both OS disk and attached data disks within the template, to happen after VM provisioning and software installation using the newer Script VM Extension for Linux 2.0 used in Quickstart templates (https://github.com/Azure/azure-quickstart-templates/issues/2340).

The template deployment succeeds and reports successful, but the encryption operation is failing to encrypt the osDisk and the data disks are not encrypted (I guess the process fails on the osDisk and doesn’t get to the data disks).

Here’s a snippet of the extension log at /var/log/azure/Microsoft.Azure.Security.AzureDiskEncryptionForLinux/0.1.0.999283/extension.log (I can provide the full one if needed)

2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] op: EnableEncryptionOSVolume
2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] status: error
2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] code: 19
2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] msg: Failed to encrypt OS volume with error: Attempt #1 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32
2017/02/03 00:16:54 stdout:
2017/02/03 00:16:54
2017/02/03 00:16:54 stderr:
2017/02/03 00:16:54 umount: /oldroot: target is busy
2017/02/03 00:16:54         (In some cases useful info about processes that
2017/02/03 00:16:54          use the device is found by lsof(8) or fuser(1).)
2017/02/03 00:16:54 , stack trace: Traceback (most recent call last):
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 166, in start_encryption
2017/02/03 00:16:54     self.enter_unmount_oldroot()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 222, in trigger
2017/02/03 00:16:54     return self.machine.process(f)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 526, in process
2017/02/03 00:16:54     return trigger()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 247, in _trigger
2017/02/03 00:16:54     if t.execute(event):
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 148, in execute
2017/02/03 00:16:54     self._change_state(event_data)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 159, in _change_state
2017/02/03 00:16:54     event_data.machine.get_state(self.dest).enter(event_data)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 48, in enter
2017/02/03 00:16:54     event_data.machine.callback(oe, event_data)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 518, in callback
2017/02/03 00:16:54     func(*event_data.args, **event_data.kwargs)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 114, in on_enter_state
2017/02/03 00:16:54     super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/OSEncryptionStateMachine.py", line 65, in on_enter_state
2017/02/03 00:16:54     self.state_objs[self.state].enter()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py", line 134, in enter
2017/02/03 00:16:54     self.command_executor.Execute('umount /oldroot', True)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/CommandExecutor.py", line 70, in Execute
2017/02/03 00:16:54     raise Exception(msg)
2017/02/03 00:16:54 Exception: Command umount /oldroot failed with return code 32
2017/02/03 00:16:54 stdout:
2017/02/03 00:16:54
2017/02/03 00:16:54 stderr:
2017/02/03 00:16:54 umount: /oldroot: target is busy
2017/02/03 00:16:54         (In some cases useful info about processes that
2017/02/03 00:16:54          use the device is found by lsof(8) or fuser(1).)
2017/02/03 00:16:54
2017/02/03 00:16:54 , stack trace: Traceback (most recent call last):
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/handle.py", line 1522, in daemon_encrypt
2017/02/03 00:16:54     os_encryption.start_encryption()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 184, in start_encryption
2017/02/03 00:16:54     raise Exception(message)
2017/02/03 00:16:54 Exception: Attempt #1 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32
2017/02/03 00:16:54 stdout:
2017/02/03 00:16:54
2017/02/03 00:16:54 stderr:
2017/02/03 00:16:54 umount: /oldroot: target is busy
2017/02/03 00:16:54         (In some cases useful info about processes that
2017/02/03 00:16:54          use the device is found by lsof(8) or fuser(1).)

Checking the encryption status through the Azure PowerShell SDK correlates with the problem in the log:

Get-AzureRmVmDiskEncryptionStatus -ResourceGroupName "encrypted-cluster" -VMName "data-0"
Get-AzureRmVmDiskEncryptionStatus : Long running operation failed with status 'Failed'.
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension 'AzureDiskEncryptionForLinux'. Error message: "Failed to encrypt OS volume with error: Attempt #1 to unmount /oldroot failed with 
error: Command umount /oldroot failed with return code 32
stdout:
stderr:
umount: /oldroot: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
, stack trace: Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 166, in start_encryption
    self.enter_unmount_oldroot()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 222, in trigger
    return self.machine.process(f)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 526, in process
    return trigger()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 247, in _trigger
    if t.execute(event):
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 148, in execute
    self._change_state(event_data)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 159, in _change_state
    event_data.machine.get_state(self.dest).enter(event_data)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 48, in enter
    event_data.machine.callback(oe, event_data)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 518, in callback
    func(*event_data.args, **event_data.kwargs)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 114, in on_enter_state
    super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/OSEncryptionStateMachine.py", line 65, in on_enter_state
    self.state_objs[self.state].enter()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py", line 134, in enter
    self.command_executor.Execute('umount /oldroot', True)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/CommandExecutor.py", line 70, in Execute
    raise Exception(msg)
Exception: Command umount /oldroot failed with return code 32

My understanding is that it is possible to encrypt the disks on a running VM based on the examples in the quickstart templates. The data disks are RAID0ed as part of the script that installs the software.

Should the encryption happen before the VM Script extension runs and software is deployed, or if it can happen after this, is this a bug in the encryption process?

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Reactions:1
  • Comments:21

github_iconTop GitHub Comments

3reactions
vsukhincommented, Jun 30, 2017

Sorry, then. I did it because, I could not wait for response in next few months (

On Fri, Jun 30, 2017 at 4:05 PM, Philipp Singer notifications@github.com wrote:

That’s not really a solution to me because I can’t even remove the encryption.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-linux-extensions/issues/334#issuecomment-312261878, or mute the thread https://github.com/notifications/unsubscribe-auth/AFtSwqx9-KQlif9surpMrV94OCh0g3Oiks5sJPKUgaJpZM4L16Z3 .

2reactions
vsukhincommented, Jun 30, 2017

I installed DM-Crypt, 2 hours and problem is solved

On Fri, Jun 30, 2017 at 11:15 AM, Philipp Singer notifications@github.com wrote:

I did encrypt two different vms, and for both I am unable to get them up again.

Checking the log, it appears to end in the error ‘’‘umount: /oldroot/mnt: not mounted’‘’.

Any ideas?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-linux-extensions/issues/334#issuecomment-312205438, or mute the thread https://github.com/notifications/unsubscribe-auth/AFtSwugB7Z4T0089xnGiIMlKRekR834Wks5sJK6agaJpZM4L16Z3 .

Read more comments on GitHub >

github_iconTop Results From Across the Web

Unable to unlock encrypted Ubuntu 16.04 - boot
I checked the encrypted option when I installed Ubuntu 16.04 yesterday just to explore other features. The installation is very fresh.
Read more >
ubuntu 16.04 self-signed cert versus let's encrypt error
I installed ubuntu 16.04 on my Droplet, and put in the LAMP stack. I manage a wordpress site, so I added that. Wordpress...
Read more >
ManualFullSystemEncryption/Troubleshooting
Still failing to fix the boot? This document belongs to Manual Full System Encryption (with Extras).
Read more >
Bug #1817689 “[16.04.6 Desktop] Cannot log in after ...
Ubuntu Desktop 16.04.6 20190222 Test Case Do an entire disk ... kernel does not support filename encryption user-setup: ERROR: Could not add ...
Read more >
openssl crypt/decrypt from ubuntu 16.04 to Centos 7.3 OK, but ...
But When I encrypt from Ubuntu 18.04 and Decrypt on a Centos 7.3 : I get this error : bad decrypt 140503464642464:error:06065064:digital ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found