Huge extension.log files from Microsoft.OSTCExtensions.LinuxDiagnostic | mdsd-lde

On multiple Ubuntu xenial hosts in private-peered Azure, behind a proxy, we are seeing excessive logging to extension.log.

Besides the root cause(s) of that, there seems to be no way to turn off the verbose logging, and there seems to be no related logrotate script.

$ sudo find /var/log/ -size +102400k -exec ls -ltrh {} \;
# -rw-r--r-- 1 root root 1.5G Mar  5 03:13 /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9027/extension.log
# -rw-r--r-- 1 root root 16G Jun 26 23:01 /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9029/extension.log
# -rw-r--r-- 1 root root 166M Jun 26 23:00 /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.4.60.2/extension.log

This is despite:

$ grep -i "log"  /etc/waagent.conf
# Enable verbose logging (y|n)
Logs.Verbose=n

There are a couple of errors in there (but that’s not my primary point here):

$ sudo tail -f /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9029/extension.log
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd ps -ef | grep mysql | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError.  Error Code is 1
2018/07/19 19:24:52 ERROR:CalledProcessError.  Command string was ps -ef | grep mysql | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError.  Command result was
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] Return 1:
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd ps -ef | grep -E 'httpd|apache2' | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError.  Error Code is 1
2018/07/19 19:24:52 ERROR:CalledProcessError.  Command string was ps -ef | grep -E 'httpd|apache2' | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError.  Command result was
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] Return 1:
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd /opt/omi/bin/service_control is-running
2018/07/19 19:24:52 ERROR:CalledProcessError.  Error Code is 1
2018/07/19 19:24:52 ERROR:CalledProcessError.  Command string was /opt/omi/bin/service_control is-running
2018/07/19 19:24:52 ERROR:CalledProcessError.  Command result was
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] Return 1:
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/bin/mdsd -v -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml
...
2018/07/19 19:27:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/bin/mdsd -v -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml
...
2018/07/19 19:28:12 ERROR:CalledProcessError.  Error Code is 1
2018/07/19 19:28:12 ERROR:CalledProcessError.  Command string was /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/bin/mdsd -v -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml
2018/07/19 19:28:12 ERROR:CalledProcessError.  Command result was 2018-07-19T19:27:52.4373460Z: Not all GCS env vars are defined. Missing 5: MONITORING_GCS_ENVIRONMENT MONITORING_GCS_ACCOUNT MONITORING_GCS_REGION MONITORING_GCS_CERT_CERTFILE MONITORING_GCS_CERT_KEYFILE. GCS won't be used.
2018/07/19 19:28:12 ERROR:Parse reported these messages:
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(3) Error: Storage credential validation for table storage failed: Error resolving address
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(32) Fatal: AddMonikerEventInfo() failed: No default credential is found.
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(38) Error: No default credentials were defined
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(43) Error: No default credentials were defined

[UnwashedMeme]

I know Azure wants to sell us more storage space but this is a bit ridiculous:

$ sudo ls -hl /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux 
total 2.5G
-rw-r--r-- 1 root root  71K Oct  7 18:39 extension.log
-rw-r--r-- 1 root root 2.5G Oct  7 18:51 watcher.log

I’m with OP that this package needs logrotate configs

[sbonds]

Thanks for the reminder-- I found that one later but never updated this issue. Here’s the complete list of the others I’ve added to log rotation (in addition to my earlier comment):

/var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/watcher.log
/var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/extension.log
/var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/CommandExecution.log

I don’t think these files are from the agent corresponding to the GitHub repo hosting this issue, but as end users we don’t really care which agent all the things filling up /var come from. Microsoft has a ton of agents that need to have their logs managed.