Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ubuntu 18.04 Custom script extension download failure

See original GitHub issue

Azure Linux Agent Version: 2.2.20

We are observing failures when running a custom script extension on an Ubuntu 18.04 VMSS deployment. It looks like for some reason the CustomScript handler is not correctly downloading all of the custom script files, and then seems to crash. This seems to happen for ~50% of our VMs (so quite a significant impact)

waagent.log contents:

2018/08/08 15:52:58.307908 INFO Azure Linux Agent Version:2.2.20
2018/08/08 15:52:58.372995 INFO OS: ubuntu 18.04
2018/08/08 15:52:58.421229 INFO Python: 3.6.5
2018/08/08 15:52:58.473775 INFO Run daemon
2018/08/08 15:52:58.547468 INFO Clean protocol
2018/08/08 15:52:58.633181 INFO Running CloudInit provisioning handler
2018/08/08 15:52:58.733407 INFO Detect protocol endpoints
2018/08/08 15:52:58.779756 INFO Clean protocol
2018/08/08 15:52:58.851137 INFO WireServer endpoint is not found. Rerun dhcp handler
2018/08/08 15:52:58.976191 INFO Test for route to 168.63.129.16
2018/08/08 15:52:59.032851 INFO Route to 168.63.129.16 exists
2018/08/08 15:52:59.077725 INFO Wire server endpoint:168.63.129.16
2018/08/08 15:52:59.133349 INFO Fabric preferred wire protocol version:2015-04-05
2018/08/08 15:52:59.180635 INFO Wire protocol version:2012-11-30
2018/08/08 15:52:59.212024 WARNING Server preferred version:2015-04-05
2018/08/08 15:52:59.853273 INFO ssh host key found at: /etc/ssh/ssh_host_rsa_key.pub
2018/08/08 15:52:59.890868 INFO Thumbprint obtained from : /etc/ssh/ssh_host_rsa_key.pub
2018/08/08 15:52:59.923015 INFO Finished provisioning
2018/08/08 15:53:00.998429 INFO Event: name=WALinuxAgent, op=Provision, message=Provision succeed, duration=2365
2018/08/08 15:53:01.034530 INFO RDMA capabilities are not enabled, skipping
2018/08/08 15:53:01.060091 INFO Installed Agent WALinuxAgent-2.2.20 is the most current agent
2018/08/08 15:53:01.399788 INFO Agent WALinuxAgent-2.2.20 is running as the goal state agent
2018/08/08 15:53:01.459401 INFO Wire server endpoint:168.63.129.16
2018/08/08 15:53:01.510526 INFO Start env monitor service.
2018/08/08 15:53:01.541595 INFO Configure routes
2018/08/08 15:53:01.570539 INFO Gateway:None
2018/08/08 15:53:01.596773 INFO Routes:None
2018/08/08 15:53:01.621382 WARNING Hostname record does not exist, creating [/var/lib/waagent/published_hostname] with hostname [4ddd791e44d04b1092dde6234815a22d000000]
2018/08/08 15:53:01.748123 INFO WALinuxAgent-2.2.20 running as process 1443
2018/08/08 15:53:01.770585 INFO Wire server endpoint:168.63.129.16
2018/08/08 15:53:01.783368 INFO Event: name=WALinuxAgent, op=HeartBeat, message=1;0;5B9B1F89-C999-4D6F-8465-4867DD274482;0, duration=0
2018/08/08 15:53:01.805892 INFO Event: name=WALinuxAgent, op=Partition, message=80, duration=0
2018/08/08 15:53:01.891734 INFO Event: name=WALinuxAgent, op=AutoUpdate, message=, duration=0
2018/08/08 15:53:01.922647 INFO Wire server endpoint:168.63.129.16
2018/08/08 15:53:02.709913 INFO Event: name=WALinuxAgent, op=Install, message=Agent WALinuxAgent-2.2.20 downloaded successfully, duration=0
2018/08/08 15:53:02.852546 INFO Event: name=WALinuxAgent, op=Install, message=Agent WALinuxAgent-2.2.25 downloaded successfully, duration=0
2018/08/08 15:53:02.919198 INFO Successfully added Azure fabric firewall rules
2018/08/08 15:53:02.947997 INFO Firewall rules:
Chain INPUT (policy ACCEPT 7 packets, 4295 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 10 packets, 1602 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            168.63.129.16        owner UID match 0
       0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            168.63.129.16        ctstate INVALID,NEW

2018/08/08 15:53:03.070080 INFO Event: name=WALinuxAgent, op=Install, message=Agent WALinuxAgent-2.2.26 downloaded successfully, duration=0
2018/08/08 15:53:03.223559 INFO Event: name=WALinuxAgent, op=Firewall, message=, duration=0
2018/08/08 15:53:03.255201 INFO Set block dev timeout: sdb with timeout: 300
2018/08/08 15:53:03.279573 INFO Set block dev timeout: sda with timeout: 300
2018/08/08 15:53:03.301549 WARNING Dhcp client is not running. 
2018/08/08 15:53:03.328918 INFO Purging disk cache, current incarnation is 1
2018/08/08 15:53:03.364339 INFO Event: name=WALinuxAgent, op=Install, message=Agent WALinuxAgent-2.2.30 downloaded successfully, duration=0
2018/08/08 15:53:03.407749 INFO Agent WALinuxAgent-2.2.20 discovered update WALinuxAgent-2.2.30 -- exiting
2018/08/08 15:53:04.121214 INFO Agent WALinuxAgent-2.2.20 launched with command 'python3 -u /usr/sbin/waagent -run-exthandlers' is successfully running
2018/08/08 15:53:04.171123 INFO Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.2.20 launched with command 'python3 -u /usr/sbin/waagent -run-exthandlers' is successfully running, duration=0
2018/08/08 15:53:04.234417 INFO Determined Agent WALinuxAgent-2.2.30 to be the latest agent
2018/08/08 15:53:04.631871 INFO ExtHandler Agent WALinuxAgent-2.2.30 is running as the goal state agent
2018/08/08 15:53:04.688057 INFO ExtHandler Wire server endpoint:168.63.129.16
2018/08/08 15:53:04.739042 INFO ExtHandler Creating cgroup directory /sys/fs/cgroup/cpu/WALinuxAgent
2018/08/08 15:53:04.772347 INFO ExtHandler Creating cgroup directory /sys/fs/cgroup/memory/WALinuxAgent
2018/08/08 15:53:04.809014 INFO ExtHandler Start env monitor service.
2018/08/08 15:53:04.843413 INFO ExtHandler Configure routes
2018/08/08 15:53:04.863677 INFO ExtHandler Gateway:None
2018/08/08 15:53:04.882956 INFO ExtHandler Routes:None
2018/08/08 15:53:04.913509 INFO ExtHandler Wire server endpoint:168.63.129.16
2018/08/08 15:53:04.936989 INFO ExtHandler Route table: [{"Iface": "eth0", "Destination": "0.0.0.0", "Gateway": "10.0.0.1", "Mask": "0.0.0.0", "Flags": "0x0003", "Metric": "100"},{"Iface": "eth0", "Destination": "10.0.0.0", "Gateway": "0.0.0.0", "Mask": "255.255.255.0", "Flags": "0x0001", "Metric": "0"},{"Iface": "eth0", "Destination": "168.63.129.16", "Gateway": "10.0.0.1", "Mask": "255.255.255.255", "Flags": "0x0007", "Metric": "100"},{"Iface": "eth0", "Destination": "169.254.169.254", "Gateway": "10.0.0.1", "Mask": "255.255.255.255", "Flags": "0x0007", "Metric": "100"}]
2018/08/08 15:53:04.937545 INFO ExtHandler WALinuxAgent-2.2.30 running as process 1496
2018/08/08 15:53:05.121710 INFO ExtHandler Successfully added Azure fabric firewall rules
2018/08/08 15:53:05.152640 INFO ExtHandler Initial NIC state: [{ "name": "lo", "link": "<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 " }, { "name": "eth0", "link": "<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000\    link/ether 00:0d:3a:4e:74:3c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 64 numrxqueues 64 gso_max_size 62780 gso_max_segs 65535 " }]
2018/08/08 15:53:05.182745 INFO ExtHandler Wire server endpoint:168.63.129.16
2018/08/08 15:53:05.345866 INFO ExtHandler Firewall rules:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            168.63.129.16        owner UID match 0
       0        0 DROP       tcp  --  *      *       0.0.0.0/0            168.63.129.16        ctstate INVALID,NEW

2018/08/08 15:53:05.527690 INFO ExtHandler Wire server endpoint:168.63.129.16
2018/08/08 15:53:05.698976 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 15:53:05.738652 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: notinstalled
2018/08/08 15:53:05.903911 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Initialize extension directory
2018/08/08 15:53:05.939466 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 15:53:05.971934 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Install extension [bin/custom-script-shim install]
2018/08/08 15:53:06.012782 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 15:53:06.050835 INFO ExtHandler Creating cgroup directory /sys/fs/cgroup/cpu/WALinuxAgent/Microsoft.Azure.Extensions.CustomScript
2018/08/08 15:53:06.092993 INFO ExtHandler Creating cgroup directory /sys/fs/cgroup/memory/WALinuxAgent/Microsoft.Azure.Extensions.CustomScript
2018/08/08 15:53:06.131175 INFO ExtHandler Now tracking cgroup Microsoft.Azure.Extensions.CustomScript
2018/08/08 15:53:07.218445 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 15:53:07.306303 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 15:53:09.477425 INFO ExtHandler Wire server endpoint:168.63.129.16
2018/08/08 15:53:09.504402 INFO ExtHandler ProcessGoalState completed [incarnation 1; 3976 ms]
2018/08/08 15:58:05.745241 INFO ExtHandler After updating cgroup telemetry, tracking [Agents+Extensions] [Microsoft.Azure.Extensions.CustomScript] [walinuxagent.service]
2018/08/08 16:08:05.069224 INFO Agent WALinuxAgent-2.2.30 launched with command 'python3 -u bin/WALinuxAgent-2.2.30-py2.7.egg -run-exthandlers' is successfully running
2018/08/08 16:08:05.121555 INFO Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.2.30 launched with command 'python3 -u bin/WALinuxAgent-2.2.30-py2.7.egg -run-exthandlers' is successfully running, duration=0
2018/08/08 16:29:37.903262 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 16:29:37.937900 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 16:29:37.976683 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 16:29:38.011735 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 16:29:38.052106 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 16:29:39.156029 INFO ExtHandler ProcessGoalState completed [incarnation 2; 1388 ms]
2018/08/08 16:54:42.731638 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 16:54:42.768088 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 16:54:42.810615 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 16:54:42.843480 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 16:54:42.881186 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 16:54:43.979935 INFO ExtHandler ProcessGoalState completed [incarnation 3; 1394 ms]
2018/08/08 17:03:41.663160 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 17:03:41.698428 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 17:03:41.732955 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 17:03:41.773793 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 17:03:41.820681 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 17:03:42.899846 INFO ExtHandler ProcessGoalState completed [incarnation 4; 1377 ms]
2018/08/08 17:25:16.333239 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 17:25:16.371490 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 17:25:16.409047 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 17:25:16.456151 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 17:25:16.503177 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 17:25:17.594332 INFO ExtHandler ProcessGoalState completed [incarnation 5; 1416 ms]
2018/08/08 18:04:23.054734 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 18:04:23.105767 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 18:04:23.139587 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 18:04:23.180554 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 18:04:23.219200 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 18:04:24.303831 INFO ExtHandler ProcessGoalState completed [incarnation 6; 1382 ms]
2018/08/08 18:50:45.737962 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 18:50:45.776148 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 18:50:45.810379 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 18:50:45.844163 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 18:50:45.884376 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 18:50:46.991078 INFO ExtHandler ProcessGoalState completed [incarnation 7; 1425 ms]
2018/08/08 19:00:35.435938 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 19:00:35.506313 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 19:00:35.546872 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 19:00:35.578633 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 19:00:35.628124 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 19:00:36.730073 INFO ExtHandler ProcessGoalState completed [incarnation 8; 1455 ms]
2018/08/08 19:03:56.853196 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 19:03:56.885010 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 19:03:56.921391 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 19:03:56.956930 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 19:03:56.998789 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 19:03:58.116365 INFO ExtHandler ProcessGoalState completed [incarnation 9; 1407 ms]
2018/08/08 19:25:23.798640 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Target handler state: enabled
2018/08/08 19:25:23.843711 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] [Enable] current handler state is: enabled
2018/08/08 19:25:23.882338 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Update settings file: 1.settings
2018/08/08 19:25:23.917233 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.6] Enable extension [bin/custom-script-shim enable]
2018/08/08 19:25:23.959355 INFO ExtHandler Move process 1496 into cgroups for extension Microsoft.Azure.Extensions.CustomScript
2018/08/08 19:25:25.043099 INFO ExtHandler ProcessGoalState completed [incarnation 10; 1396 ms]

customscript handler.log contents:

+ /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension install
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 event=start
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 status="not reported for operation (by design)"
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 event="migrate to mrseq" error="Can't find out seqnum from /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status, not enough files."
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 event="created data dir" path=/var/lib/waagent/custom-script
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 event=installed
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 status="not reported for operation (by design)"
time=2018-08-08T15:53:06Z version=v2.0.6/git@1008306-clean operation=install seq=1 event=end
Writing a placeholder status file indicating progress before forking: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="seqnum saved" path=mrseq
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="reading configuration"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="read configuration"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="validating json schema"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="json schema valid"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="parsing configuration json"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="parsed configuration json"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="validating configuration logically"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="validated configuration"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="creating output directory" path=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="created output directory"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 files=10
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=0 event="download start"
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=0 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:07Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=1 event="download start"
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=1 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=2 event="download start"
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=2 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=3 event="download start"
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=3 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=4 event="download start"
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=4 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:08Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=5 event="download start"
time=2018-08-08T15:53:10Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=5 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:10Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=6 event="download start"
time=2018-08-08T15:53:12Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=6 event="download complete" output=/var/lib/waagent/custom-script/download/1
time=2018-08-08T15:53:12Z version=v2.0.6/git@1008306-clean operation=enable seq=1 file=7 event="download start"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T16:29:38Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T16:29:38Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T16:29:38Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T16:29:38Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T16:54:42Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T16:54:42Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T16:54:42Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T16:54:42Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T17:03:41Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T17:03:41Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T17:03:41Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T17:03:41Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T17:25:16Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T17:25:16Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T17:25:16Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T17:25:16Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T18:04:23Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T18:04:23Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T18:04:23Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T18:04:23Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T18:50:45Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T18:50:45Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T18:50:45Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T18:50:45Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T19:00:35Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T19:00:35Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T19:00:35Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T19:00:35Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T19:03:57Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T19:03:57Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T19:03:57Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T19:03:57Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"
Not writing a placeholder status file, already exists: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/status/1.status
+ nohup /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.6/bin/custom-script-extension enable
time=2018-08-08T19:25:24Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=start
time=2018-08-08T19:25:24Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=pre-check
time=2018-08-08T19:25:24Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event="comparing seqnum" path=mrseq
time=2018-08-08T19:25:24Z version=v2.0.6/git@1008306-clean operation=enable seq=1 event=exit message="this script configuration is already processed, will not run again"

The key thing is that there are 10 files to download in handler.log but it never actually makes it to file id=9 – it seems to fail during the download of file id 7.

Interestingly, this manifests itself strangely in the Azure portal, as all the VMs having this issue end up stuck in “Creating (Running)” state forever. Eventually the deployment of the VMSS fails with “Operation abandoned due to internal error. Please try again later.”

Strangely, we use basically the same custom script extension (10 files as well) on other OS’s (16.04, 14.04, Debian, etc) and things seem to work fine. I wonder if it has something to do with the fact the default Python version in Ubuntu18.04 is Python 3?

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
matthchrcommented, Aug 13, 2018

It looks like the 2.2.26 rollback fixed this. I’m closing this.

0reactions
johnwhumphreyscommented, Jan 2, 2019

Nevermind… this worked for me (I was using Centos 7.5 but suspect the same issue).

yum -y downgrade WALinuxAgent-2.2.26

Read more comments on GitHub >

github_iconTop Results From Across the Web

http request failed: GET dial tcp x.x.x.x:443 i/o timeout ... - GitHub
I am deploying the Custom Script Extension to a VMSS running Ubuntu 18.04. I configured the custom script extension with the ...
Read more >
apt-get update not working as azure custom script extension
I'm deploying linux ubuntu 16.04 LTS VMSS using Azure ARM template with custom script extension. Content of customscript.sh: apt-get update apt ...
Read more >
Azure Custom Script Extension On Linux VM and VMSS
Recently, there was a post about Custom Script Extension for Windows, however, there's a similar functionality for Linux VMs.
Read more >
Run Custom Script Extension on Linux VMs in Azure
The Custom Script Extension Version 2 downloads and runs scripts on Azure virtual machines (VMs). This extension is useful for ...
Read more >
Manually install the Deep Security Agent
Install the agent on a Microsoft Azure VM · Generate and run a deployment script · Add a custom script extension to an...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

WALinuxAgent 2.2.26 - high IO load on Ubuntu 16.04

Next page

Issues with waagent.conf and swap