Encryption failing on Ubuntu 16.04

I’m building Azure Disk Encryption into an ARM template and have configured encryption of both OS disk and attached data disks within the template, to happen after VM provisioning and software installation using the newer Script VM Extension for Linux 2.0 used in Quickstart templates (https://github.com/Azure/azure-quickstart-templates/issues/2340).

The template deployment succeeds and reports successful, but the encryption operation is failing to encrypt the osDisk and the data disks are not encrypted (I guess the process fails on the osDisk and doesn’t get to the data disks).

Here’s a snippet of the extension log at /var/log/azure/Microsoft.Azure.Security.AzureDiskEncryptionForLinux/0.1.0.999283/extension.log (I can provide the full one if needed)

2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] op: EnableEncryptionOSVolume
2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] status: error
2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] code: 19
2017/02/03 00:16:54 [Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: [StatusReport (0)] msg: Failed to encrypt OS volume with error: Attempt #1 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32
2017/02/03 00:16:54 stdout:
2017/02/03 00:16:54
2017/02/03 00:16:54 stderr:
2017/02/03 00:16:54 umount: /oldroot: target is busy
2017/02/03 00:16:54         (In some cases useful info about processes that
2017/02/03 00:16:54          use the device is found by lsof(8) or fuser(1).)
2017/02/03 00:16:54 , stack trace: Traceback (most recent call last):
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 166, in start_encryption
2017/02/03 00:16:54     self.enter_unmount_oldroot()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 222, in trigger
2017/02/03 00:16:54     return self.machine.process(f)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 526, in process
2017/02/03 00:16:54     return trigger()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 247, in _trigger
2017/02/03 00:16:54     if t.execute(event):
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 148, in execute
2017/02/03 00:16:54     self._change_state(event_data)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 159, in _change_state
2017/02/03 00:16:54     event_data.machine.get_state(self.dest).enter(event_data)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 48, in enter
2017/02/03 00:16:54     event_data.machine.callback(oe, event_data)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 518, in callback
2017/02/03 00:16:54     func(*event_data.args, **event_data.kwargs)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 114, in on_enter_state
2017/02/03 00:16:54     super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/OSEncryptionStateMachine.py", line 65, in on_enter_state
2017/02/03 00:16:54     self.state_objs[self.state].enter()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py", line 134, in enter
2017/02/03 00:16:54     self.command_executor.Execute('umount /oldroot', True)
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/CommandExecutor.py", line 70, in Execute
2017/02/03 00:16:54     raise Exception(msg)
2017/02/03 00:16:54 Exception: Command umount /oldroot failed with return code 32
2017/02/03 00:16:54 stdout:
2017/02/03 00:16:54
2017/02/03 00:16:54 stderr:
2017/02/03 00:16:54 umount: /oldroot: target is busy
2017/02/03 00:16:54         (In some cases useful info about processes that
2017/02/03 00:16:54          use the device is found by lsof(8) or fuser(1).)
2017/02/03 00:16:54
2017/02/03 00:16:54 , stack trace: Traceback (most recent call last):
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/handle.py", line 1522, in daemon_encrypt
2017/02/03 00:16:54     os_encryption.start_encryption()
2017/02/03 00:16:54   File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 184, in start_encryption
2017/02/03 00:16:54     raise Exception(message)
2017/02/03 00:16:54 Exception: Attempt #1 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32
2017/02/03 00:16:54 stdout:
2017/02/03 00:16:54
2017/02/03 00:16:54 stderr:
2017/02/03 00:16:54 umount: /oldroot: target is busy
2017/02/03 00:16:54         (In some cases useful info about processes that
2017/02/03 00:16:54          use the device is found by lsof(8) or fuser(1).)

Checking the encryption status through the Azure PowerShell SDK correlates with the problem in the log:

Get-AzureRmVmDiskEncryptionStatus -ResourceGroupName "encrypted-cluster" -VMName "data-0"
Get-AzureRmVmDiskEncryptionStatus : Long running operation failed with status 'Failed'.
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension 'AzureDiskEncryptionForLinux'. Error message: "Failed to encrypt OS volume with error: Attempt #1 to unmount /oldroot failed with 
error: Command umount /oldroot failed with return code 32
stdout:
stderr:
umount: /oldroot: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
, stack trace: Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 166, in start_encryption
    self.enter_unmount_oldroot()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 222, in trigger
    return self.machine.process(f)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 526, in process
    return trigger()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 247, in _trigger
    if t.execute(event):
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 148, in execute
    self._change_state(event_data)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 159, in _change_state
    event_data.machine.get_state(self.dest).enter(event_data)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 48, in enter
    event_data.machine.callback(oe, event_data)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py", line 518, in callback
    func(*event_data.args, **event_data.kwargs)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py", line 114, in on_enter_state
    super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/OSEncryptionStateMachine.py", line 65, in on_enter_state
    self.state_objs[self.state].enter()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py", line 134, in enter
    self.command_executor.Execute('umount /oldroot', True)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/CommandExecutor.py", line 70, in Execute
    raise Exception(msg)
Exception: Command umount /oldroot failed with return code 32

My understanding is that it is possible to encrypt the disks on a running VM based on the examples in the quickstart templates. The data disks are RAID0ed as part of the script that installs the software.

Should the encryption happen before the VM Script extension runs and software is deployed, or if it can happen after this, is this a bug in the encryption process?

[vsukhin]

Sorry, then. I did it because, I could not wait for response in next few months (

On Fri, Jun 30, 2017 at 4:05 PM, Philipp Singer [e-mail] wrote:

That’s not really a solution to me because I can’t even remove the encryption.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-linux-extensions/issues/334#issuecomment-312261878, or mute the thread https://github.com/notifications/unsubscribe-auth/AFtSwqx9-KQlif9surpMrV94OCh0g3Oiks5sJPKUgaJpZM4L16Z3 .

[vsukhin]

I installed DM-Crypt, 2 hours and problem is solved

On Fri, Jun 30, 2017 at 11:15 AM, Philipp Singer [e-mail] wrote:

I did encrypt two different vms, and for both I am unable to get them up again.

Checking the log, it appears to end in the error ‘’‘umount: /oldroot/mnt: not mounted’‘’.

Any ideas?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-linux-extensions/issues/334#issuecomment-312205438, or mute the thread https://github.com/notifications/unsubscribe-auth/AFtSwugB7Z4T0089xnGiIMlKRekR834Wks5sJK6agaJpZM4L16Z3 .