question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

WALinuxAgent 2.2.26 - high IO load on Ubuntu 16.04

See original GitHub issue

Hi,

We have an Ubuntu 16.04.5 VM which shows high IO load even when none of our processes is running.

Output from iostat -x:


Linux 4.15.0-1023-azure (*****)   10/02/2018      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          30.71    0.05   11.19   26.92    0.00   31.13

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     3.20     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     3.24    2.59   20.38    84.53 18268.50  1597.65    66.94 3076.89   64.91 3460.13  30.23  69.46
sdb               0.00     3.64    0.04    0.09     1.36  1862.07 29863.31     0.08  637.71    5.66  887.47  13.95   0.17

Note the %util and w_await values - writes to sda have a latency of 3.5 seconds!

iotop -Poa shows only the WALinuxAgent and jbd2/sda1-8 writing to disk (our processes are down):

Total DISK READ :       0.00 B/s | Total DISK WRITE :     106.03 M/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:      87.93 M/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  364 be/3 root          0.00 B   1036.00 K  0.00 % 53.89 % [jbd2/sda1-8]
 1417 be/4 root          0.00 B      3.56 G  0.00 % 14.91 % python3 -u bin/WALinuxAgent-2.2.26-py2.7.egg -run-exthandlers
31448 be/4 root          0.00 B      0.00 B  0.00 %  2.61 % [kworker/u4:2]
 1183 be/4 syslog        0.00 B    208.00 K  0.00 %  0.76 % rsyslogd -n
 6148 be/4 root          0.00 B      0.00 B  0.00 %  0.03 % [kworker/u4:1]
 1690 be/4 omsagent      0.00 B      8.00 K  0.00 %  0.00 % ruby /opt/microsoft/omsagent/bin/omsagent-75~e51-4e30-b909-8f439eddfca8/conf/omsagent.conf
 1769 be/4 omsagent      0.00 B      4.00 K  0.00 %  0.00 % omiagent 9 10 --destdir / --providerdir /opt/omi/lib --loglevel WARNING
 1298 be/4 vnstat        0.00 B      8.00 K  0.00 %  0.00 % vnstatd -n

last 100 lines of /var/log/waagent.log:

2018/10/02 09:23:56.444878 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:24:02.191226 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:24:02.198660 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:24:04.858610 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2654
2018/10/02 09:24:04.920888 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:24:04.963983 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:24:05.002069 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:24:05 OmsAgentForLinux started to handle.
2018/10/02 09:24:05 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:24:05 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:24:06.050221 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:24:06.166199 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:24:19.049544 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:24:19.098244 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:24:21.252683 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2100
2018/10/02 09:24:21.310976 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:24:21.357628 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:24:21.409811 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:24:21 OmsAgentForLinux started to handle.
2018/10/02 09:24:21 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:24:21 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:24:22.462941 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:24:22.588928 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:24:28.392800 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:24:28.439312 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:24:30.844747 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2343
2018/10/02 09:24:30.852199 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:24:30.857982 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:24:30.863206 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:24:30 OmsAgentForLinux started to handle.
2018/10/02 09:24:30 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:24:30 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:24:31.873319 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:24:31.978853 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:24:37.899197 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:24:37.907344 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:24:40.094352 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2181
2018/10/02 09:24:40.105080 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:24:40.114729 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:24:40.118347 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:24:40 OmsAgentForLinux started to handle.
2018/10/02 09:24:40 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:24:40 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:24:41.130790 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:24:41.212059 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:24:52.689966 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:24:52.740435 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:24:54.891195 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2097
2018/10/02 09:24:54.898307 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:24:54.904349 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:24:54.909326 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:24:54 OmsAgentForLinux started to handle.
2018/10/02 09:24:54 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:24:54 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:24:55.918720 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:24:56.026496 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:25:02.039767 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:25:02.046786 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:25:12.092069 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=10040
2018/10/02 09:25:12.099285 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:25:15.735402 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:25:15.743076 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:25:15 OmsAgentForLinux started to handle.
2018/10/02 09:25:15 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:25:15 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:25:16.752356 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:25:16.879122 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:25:20.019371 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:25:20.078037 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:25:22.453753 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2320
2018/10/02 09:25:24.734801 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:25:24.743186 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:25:24.761554 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:25:24 OmsAgentForLinux started to handle.
2018/10/02 09:25:24 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:25:24 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:25:25.776197 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:25:25.883855 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:25:29.387110 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:25:29.394149 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:25:31.386648 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=1984
2018/10/02 09:25:31.448473 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:25:31.496810 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:25:31.552868 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:25:31 OmsAgentForLinux started to handle.
2018/10/02 09:25:35 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:25:35 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:25:35.599810 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:25:35.705796 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/02 09:25:45.557988 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/02 09:25:45.565194 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/02 09:25:48.041069 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=2469
2018/10/02 09:25:48.048063 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/02 09:25:48.054082 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/02 09:25:48.058950 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable extension [omsagent.py -disable]
2018/10/02 09:25:48 OmsAgentForLinux started to handle.
2018/10/02 09:25:48 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0
2018/10/02 09:25:48 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log
2018/10/02 09:25:49.068457 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/02 09:25:49.145750 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9

/var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.6.42.0/extension.log has a bunch of these:

2018/10/02 09:28:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] sequence number is 0
2018/10/02 09:28:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] setting file path is/var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0/config/0.settings
2018/10/02 09:28:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] JSON config: {"runtimeSettings": [{"handlerSettings": {"protectedSettings": "*REMOVED*", "publicSettings": {"stopOnMultipleConnections": true, "workspaceId": "*REMOVED*"}, "protectedSettingsCertThumbprint": "*REMOVED*"}}]}
2018/10/02 09:28:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Config decoded correctly.
2018/10/02 09:28:53 ERROR:CalledProcessError.  Error Code is 1
2018/10/02 09:28:53 ERROR:CalledProcessError.  Command string was /opt/microsoft/omsagent/bin/service_control disable
2018/10/02 09:28:53 ERROR:CalledProcessError.  Command result was Failed to stop omsagent-.service: Unit omsagent-.service not loaded.
2018/10/02 09:28:53 ERROR:ERROR:  disable_omsagent_service failed with result '5' on workspace *REMOVED*.
2018/10/02 09:28:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Output of command "/opt/microsoft/omsagent/bin/service_control disable":
2018/10/02 09:28:53 Failed to stop omsagent-.service: Unit omsagent-.service not loaded.
2018/10/02 09:28:53 ERROR:  disable_omsagent_service failed with result '5' on workspace *REMOVED*.
2018/10/02 09:28:53 ERROR:Disable failed with exit code 1
2018/10/02 09:28:53 ERROR:[Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable failed with exit code 1
2018/10/02 09:28:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.6.42.0] Disable,failed,1,Disable failed with exit code 1

Any help would be appreciated. Thanks

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:2
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
preetpalScommented, Oct 19, 2018

This is a terrible bug IMHO (it degrades performance and costs money). If you stop the walinuxagent service, it will still start on a reboot anyways, so the problem really isn’t mitigated. If you were to actually disable the service, you would lose the functionality the service provides even after rebooting (which is not ideal). There is no good way to mitigate this issue other than a bug fix for #1342. The service is basically unusable with this bug.

root@ubuntu1604:~# /usr/sbin/waagent -version
WALinuxAgent-2.2.20 running on ubuntu 16.04
Python: 3.5.2
Goal state agent: 2.2.20
root@ubuntu1604:~# tail -n 100 /var/log/waagent.log
2018/10/19 04:36:13.930365 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:36:13.959899 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:36:14.004299 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:36:16.331001 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:36:16 OmsAgentForLinux started to handle.
2018/10/19 04:36:16 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:36:16 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:36:17.370767 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:36:17.423357 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:36:17.488035 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:36:20.613928 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:36:20.647531 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:36:34.266506 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:36:34.309848 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:36:34.336911 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:36:34.374393 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:36:34.407730 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:36:34 OmsAgentForLinux started to handle.
2018/10/19 04:36:34 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:36:34 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:36:35.445716 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:36:35.499550 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:36:37.353117 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:36:40.465532 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:36:40.499631 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:36:52.923080 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:36:52.961355 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:36:52.991306 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:36:53.030404 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:36:53.062888 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:36:53 OmsAgentForLinux started to handle.
2018/10/19 04:36:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:36:53 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:36:54.101208 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:36:54.155795 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:36:54.205546 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:36:57.381820 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:36:57.415604 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:37:10.885286 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:37:10.925961 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:37:10.952034 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:37:10.991125 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:37:11.026538 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:37:11 OmsAgentForLinux started to handle.
2018/10/19 04:37:11 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:37:11 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:37:12.065742 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:37:12.125567 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:37:12.182848 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:37:15.294792 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:37:15.326236 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:37:29.246613 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:37:29.291577 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:37:29.318471 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:37:29.359421 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:37:29.393446 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:37:29 OmsAgentForLinux started to handle.
2018/10/19 04:37:29 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:37:29 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:37:30.429855 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:37:30.481243 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:37:30.548807 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:37:35.534963 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:37:35.567874 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:37:48.443504 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:37:48.482822 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:37:49.531979 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:37:49.568400 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:37:49.601246 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:37:49 OmsAgentForLinux started to handle.
2018/10/19 04:37:49 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:37:49 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:37:50.638130 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:37:50.687977 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:37:50.739322 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:37:53.844624 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:37:53.886246 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:38:06.860593 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:38:06.903282 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:38:07.933257 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:38:07.971268 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:38:08.004318 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:38:08 OmsAgentForLinux started to handle.
2018/10/19 04:38:08 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:38:08 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:38:09.043868 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:38:09.090831 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:38:09.137419 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
2018/10/19 04:38:14.081255 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Target handler state: enabled
2018/10/19 04:38:14.115111 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] [Enable] current handler state is: notinstalled
2018/10/19 04:38:27.044144 INFO Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=Download succeeded, duration=0
2018/10/19 04:38:27.101380 WARNING Too many files under: /var/lib/waagent/events, removing oldest
2018/10/19 04:38:27.132057 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Initialize extension directory
2018/10/19 04:38:27.171375 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Update settings file: 0.settings
2018/10/19 04:38:27.216344 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Disable extension [omsagent.py -disable]
2018/10/19 04:38:27 OmsAgentForLinux started to handle.
2018/10/19 04:38:27 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3
2018/10/19 04:38:27 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.3] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.3/extension.log
2018/10/19 04:38:28.266097 ERROR Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable, duration=0
2018/10/19 04:38:28.326854 WARNING Too many files under: /var/lib/waagent/events, removing oldest
root@ubuntu1604:~#
1reaction
maltalexcommented, Oct 3, 2018

Here’s the output from /usr/sbin/waagent -version:

WALinuxAgent-2.2.20 running on ubuntu 16.04
Python: 3.5.2
Goal state agent: 2.2.20

I’d like to point out that disk write operations on Azure cost money, so this bug doesn’t only impact performance. It also impacts the monthly bill.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to update the Azure Linux Agent on a VM - Microsoft Learn
Ubuntu. Check your current package version ... apt list --installed | grep walinuxagent ... Restart waagent service for 16.04 / 17.04.
Read more >
Azure VM Ubuntu 16.04 diks ready bytes spikes - Stack Overflow
Simplest answer to fix your VM from becoming unresponsive is to up your VM size and increase your disk performance so it can...
Read more >
cloud-init does not appear to be running error after installing ...
After a recent migration of a Linux VM (Ubuntu 16.04) from a data centre to Azure using Azure Migrate, it didn't include the...
Read more >
924463 - stretch-pu: package waagent/2.2.34-3~deb9u1
These include provisioning, user account, disk, extension and networking ... -If set, waagent will respond to load balancer probes from the ...
Read more >
Options for performance improvements on very big ...
The RAM hint is a good one, but I fear that I will only get a virgin-like filesystem performance if I totally avoid...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found