[BUG] WAAgent fills disk with log entries - error: "NameError: name 'get_python_cmd' is not defined"

On multiple Azure Linux VMs, the boot drives fill up to the point where the machine is unusable, because of the /var/log/waagent.log file becoming absolutely massive (~15GB on one machine). It seems that it cannot find a Python module, and re-attempts to run a process continuously.

Distro and WALinuxAgent details WALinuxAgent-2.3.1.1 running on centos 8.4.2105 Python: 3.6.8 Goal state agent: 2.3.1.1

Additional context This started happening on two machine over the past few days. The two machines where I’ve seen this fail to respond, because their disks are full. Reinstalling the package has no effect.

The only solutions seem to be to delete the huge log file, and either disable or remove WALinuxAgent.

The machines have other versions of Python installed at /opt/python/VERSION, e.g.:

$ ls /opt/python/
3.6.12  3.7.10  3.8.8  3.9.4  miniconda  miniconda.sh

But these versions have been present for some time, and shouldn’t cause a conflict. The default version of Python on the PATH is:

/opt/python/3.9.4/bin/python

Log file entries The log file at /var/log/waagent.log is full of hundreds of thousands of entries similar to the one below:

2021-08-23T08:00:21.856964Z INFO Daemon Daemon Installed Agent WALinuxAgent-2.3.1.1 is the most current agent
2021-08-23T08:00:21.857200Z WARNING Daemon Daemon Agent WALinuxAgent-2.3.1.1 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined
2021-08-23T08:00:21.857406Z ERROR Daemon Daemon Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.3.1.1 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/azurelinuxagent/ga/update.py", line 156, in run_latest
    cmds[0] = get_python_cmd()
NameError: name 'get_python_cmd' is not defined

I’d be grateful for any assistance.

[pfernie]

@narrieta Thanks, I think you are correct; this seems like a mis-package. The 2.3.1.1 package available from openlogic repo is located here Examining the contents of that rpm reflect a update.py that looks to be a badly patched version; it refers to get_python_cmd() but lacks a definition for it:

site-packages/azurelinuxagent/ga via 🐍 v3.9.6
❯ rg get_python_cmd update.py                                             21-08-23 15:43
156:            cmds[0] = get_python_cmd()

On investigation, on our setup that repo is only providing these spurious packages; downgrading the package to 2.2.49.2 and disabling the repo has corrected the problem here; hopefully that is useful to @capncodewash as well.

[capncodewash]

Thank you very much @narrieta and @pfernie for looking into this.

Indeed I can see the same thing as @pfernie :

$ grep -n get_python_cmd /usr/lib/python3.6/site-packages/azurelinuxagent/ga/update.py
156:            cmds[0] = get_python_cmd()

waagent --version:

$ waagent --version
WALinuxAgent-2.3.1.1 running on centos 8.4.2105
Python: 3.6.8
Goal state agent: 2.3.1.1

Package version:

$ sudo rpm -qa | grep -i WALinuxAgent
WALinuxAgent-udev-2.3.1.1-1_ol001.el8.noarch
WALinuxAgent-2.3.1.1-1_ol001.el8.noarch

I’ve reported it as a bug to the CentOS project here: https://bugs.centos.org/view.php?id=18287

In the meantime, since I don’t really use the package, I’ve just stopped and disabled the services. Hopefully the CentOS team will update the package.

I’ll close this issue.