Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Azure WireServer endpoint not found
See original GitHub issueOverview
We are seeing random VM timeouts in our k8s cluster api tests, an example is: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-capz-master-containerd-windows/1519329033286848512
We are using the UserData plugin to execute run commands and set up ssh keys. Here is an example output from a successful run. If understand correctly, everything that is executed during UserData is running during Azure’s “Creating” phase of the vm because it runs during Azures VM provisioning phase.
I’ve found that this VM is timing out during this Azure VM provisioning phase. When the time out occurs the response from azure is the following and the vm never finishes provision so we are able to get logs.
{\"code\":\"OSProvisioningTimedOut\",\"message\":\"OS Provisioning for VM 'capz-conf-jjd9j' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. For details on how to check current provisioning state of Windows VMs, refer to https://aka.ms/WindowsVMLifecycle and Linux VMs, refer to https://aka.ms/LinuxVMLifecycle.\"}]}}",
what is happening
Was able to get a vm that was in this state and found:
2022-04-28 15:24:14.315 4296 DEBUG cloudbaseinit.metadata.services.azureservice [-] Azure WireServer endpoint not found load C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\metadata\services\azureservice.py:457
2022-04-28 15:24:14.316 4296 ERROR cloudbaseinit.init [-] No metadata service found: cloudbaseinit.exception.MetadataNotFoundException: No available service found
2022-04-28 15:24:14.316 4296 INFO cloudbaseinit.init [-] Plugins execution done
2022-04-28 15:24:14.316 4296 INFO cloudbaseinit.init [-] Stopping Cloudbase-Init service
2022-04-28 15:24:17.316 4296 DEBUG cloudbaseinit.osutils.windows [-] Stopping service cloudbase-init stop_service C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1140
This looks to be triggering from https://github.com/cloudbase/cloudbase-init/blob/39e2f6faf4f8062394bc61ccd0a833e7c8b76917/cloudbaseinit/metadata/services/azureservice.py#L452-L458
which is calling a failing to reach the endpoint over dhcp:
It seems because of this exception the service is never setting the failed/succeed of the provisioning step: https://github.com/cloudbase/cloudbase-init/blob/39e2f6faf4f8062394bc61ccd0a833e7c8b76917/cloudbaseinit/metadata/services/azureservice.py#L217-L224
questions
The metadata service ip is well known at 169.254.169.254 why is it querying for it via dhcp? Also it get the wireservice ip of 168.63.129.16 in the success case.
If it can’t get it via dhcp, then could if fall back to the well known ips for last try?
When this exception happens shouldn’t the failed status be set on the VM provisioning? This way it doesn’t get stuck in the Azure Creating step.
Thanks for the help!
additional info
full logs:
cloudbase-init.log
``` 2022-04-28 15:14:06.975 4244 WARNING cloudbaseinit.utils.log [-] Serial port: COM1,115200,N,8 could not be opened: serial.serialutil.SerialException: could not open port 'COM1': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2) 2022-04-28 15:14:06.976 4244 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:07.190 4244 DEBUG cloudbaseinit.osutils.windows [-] Checking if service exists: cloudbase-init check_service_exists C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1092 2022-04-28 15:14:07.193 4244 DEBUG cloudbaseinit.osutils.windows [-] Getting service username: cloudbase-init get_service_username C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1225 2022-04-28 15:14:07.193 4244 DEBUG cloudbaseinit.osutils.windows [-] Resetting password for service user: .\cloudbase-init reset_service_password C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1248 2022-04-28 15:14:07.240 4244 DEBUG cloudbaseinit.osutils.windows [-] Setting service credentials: cloudbase-init set_service_credentials C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1208 2022-04-28 15:14:07.250 4244 INFO cloudbaseinit.init [-] Respawning current process with updated credentials. 2022-04-28 15:14:07.250 4244 DEBUG cloudbaseinit.osutils.windows [-] Creating logon session for user: .\cloudbase-init create_user_logon_session C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:649 2022-04-28 15:14:07.255 4244 DEBUG cloudbaseinit.osutils.windows [-] Executing process as user, command line: ['C:\\Program Files\\Cloudbase Solutions\\Cloudbase-Init\\Python\\Scripts\\cloudbase-init', '--config-file', 'C:\\Program Files\\Cloudbase Solutions\\Cloudbase-Init\\conf\\cloudbase-init.conf', '--noreset_service_password'] execute_process_as_user C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:703 2022-04-28 15:14:07.791 4296 WARNING cloudbaseinit.utils.log [-] Serial port: COM1,115200,N,8 could not be opened: serial.serialutil.SerialException: could not open port 'COM1': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2) 2022-04-28 15:14:07.791 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:07.997 4296 INFO cloudbaseinit.init [-] Cloudbase-Init version: 1.1.2 2022-04-28 15:14:07.998 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.windows.createuser.CreateUserPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.085 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.171 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.224 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.userdata.UserDataPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.346 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.ephemeraldisk.EphemeralDiskPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.369 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.windows.azureguestagent.AzureGuestAgentPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.391 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.mtu.MTUPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.409 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.457 4296 INFO cloudbaseinit.init [-] Executing plugins for stage 'PRE_NETWORKING': 2022-04-28 15:14:08.458 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.windows.createuser.CreateUserPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.458 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.458 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.459 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.userdata.UserDataPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.459 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.ephemeraldisk.EphemeralDiskPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.459 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.windows.azureguestagent.AzureGuestAgentPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.460 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.mtu.MTUPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.460 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.460 4296 INFO cloudbaseinit.init [-] Executing plugins for stage 'PRE_METADATA_DISCOVERY': 2022-04-28 15:14:08.460 4296 INFO cloudbaseinit.init [-] Executing plugin 'MTUPlugin' 2022-04-28 15:14:08.461 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.474 4296 DEBUG cloudbaseinit.plugins.common.mtu [-] Could not obtain the MTU configuration via DHCP for interface "60:45:BD:0F:3D:F9" execute C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\plugins\common\mtu.py:46 2022-04-28 15:14:08.475 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.metadata.services.azureservice.AzureService' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:14:08.800 4296 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:24:14.315 4296 DEBUG cloudbaseinit.metadata.services.azureservice [-] Azure WireServer endpoint not found load C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\metadata\services\azureservice.py:457 2022-04-28 15:24:14.316 4296 ERROR cloudbaseinit.init [-] No metadata service found: cloudbaseinit.exception.MetadataNotFoundException: No available service found 2022-04-28 15:24:14.316 4296 INFO cloudbaseinit.init [-] Plugins execution done 2022-04-28 15:24:14.316 4296 INFO cloudbaseinit.init [-] Stopping Cloudbase-Init service 2022-04-28 15:24:17.316 4296 DEBUG cloudbaseinit.osutils.windows [-] Stopping service cloudbase-init stop_service C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1140 ```
cloudbase-init-unattend.log
``` 2022-04-28 15:13:46.682 1476 WARNING cloudbaseinit.utils.log [-] Serial port: COM1,115200,N,8 could not be opened: serial.serialutil.SerialException: could not open port 'COM1': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2) 2022-04-28 15:13:46.682 1476 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:13:49.318 1476 DEBUG cloudbaseinit.osutils.windows [-] Checking if service exists: cloudbase-init check_service_exists C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1092 2022-04-28 15:13:49.318 1476 DEBUG cloudbaseinit.osutils.windows [-] Getting service username: cloudbase-init get_service_username C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1225 2022-04-28 15:13:49.318 1476 DEBUG cloudbaseinit.osutils.windows [-] Resetting password for service user: .\cloudbase-init reset_service_password C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1248 2022-04-28 15:13:49.365 1476 DEBUG cloudbaseinit.osutils.windows [-] Setting service credentials: cloudbase-init set_service_credentials C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:1208 2022-04-28 15:13:49.381 1476 DEBUG cloudbaseinit.init [-] No need to respawn process. Current user: capz-conf-r7bm9$. Service user: cloudbase-init _reset_service_password_and_respawn C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\init.py:141 2022-04-28 15:13:49.381 1476 INFO cloudbaseinit.init [-] Cloudbase-Init version: 1.1.2 2022-04-28 15:13:49.381 1476 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.mtu.MTUPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:13:49.428 1476 INFO cloudbaseinit.init [-] Executing plugins for stage 'PRE_NETWORKING': 2022-04-28 15:13:49.428 1476 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.mtu.MTUPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:13:49.428 1476 INFO cloudbaseinit.init [-] Executing plugins for stage 'PRE_METADATA_DISCOVERY': 2022-04-28 15:13:49.428 1476 INFO cloudbaseinit.init [-] Executing plugin 'MTUPlugin' 2022-04-28 15:13:49.428 1476 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:13:49.428 1476 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.metadata.services.base.EmptyMetadataService' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:13:49.475 1476 INFO cloudbaseinit.init [-] Metadata service loaded: 'EmptyMetadataService' 2022-04-28 15:13:49.475 1476 DEBUG cloudbaseinit.init [-] Instance id: None configure_host C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\init.py:202 2022-04-28 15:13:49.475 1476 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.plugins.common.mtu.MTUPlugin' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2022-04-28 15:13:49.475 1476 INFO cloudbaseinit.init [-] Executing plugins for stage 'MAIN': 2022-04-28 15:13:49.475 1476 INFO cloudbaseinit.init [-] Plugins execution done ```
Issue Analytics
- State:
- Created a year ago
- Comments:27 (8 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Hello @jsturtevant @marosset,
The wireserver patch has been merged, also a few other patches. We are switching to Python 3.10, I will update here once the new MSI containing the patches and Py3.10 is released.
https://github.com/cloudbase/cloudbase-init-installer/pull/28
Thank you, Adrian Vladu
Hello,
There is work in progress to get the cloudbase-init patch merged here:
https://review.opendev.org/c/x/cloudbase-init/+/856306
If you can take a look and give a +1, it would be great. I expect no other functional additions to it, maybe some nitpicking on the code review, which should not take long.
Thank you, Adrian Vladu