mqttwarn is eating up all memory

I’m using mqttwarn for a while now. It works great, but I’m running into an recurring issue after ~4 days of uptime. mqttwarn is eating up all memory available on my RPi2. When I start it, it’s around 2.2%, after a few days it’s consuming ~75% and the load on the RPi is increasing to 25. It tooked a while to figure this out, as the RPi seemed to be dead after a while.

The same issue happens on a RPi4 with a similar configuration. I use mqttwarn to trigger the Icinga 2 API upon MQTT events.

mqttwarn.ini

; ------------------------------------------
;             Base configuration
; ------------------------------------------

[defaults]

; ----
; MQTT
; ----

hostname     = 'localhost'
port         = 1883
username     = 'mqttwarn'
password     = ''
clientid     = 'mqttwarn'
lwt          = 'clients/mqttwarn'
skipretained = False
cleansession = False
protocol     = 3

; -------
; Logging
; -------

; Send log output to STDERR
logfile   = 'stream://sys.stdout'

; one of: CRITICAL, DEBUG, ERROR, INFO, WARN
loglevel  = ERROR

; --------
; Services
; --------

; path to file containing self-defined functions for formatmap, alldata, and datamap
functions = '/opt/service/mqttwarn/functions.py'

; name the service providers you will be using.
launch    = log, icinga2

; -------
; Targets
; -------
[config:log]
targets = {
    'debug'  : [ 'debug' ],
    'info'   : [ 'info' ],
    'warn'   : [ 'warn' ],
    'crit'   : [ 'crit' ],
    'error'  : [ 'error' ]
  }

[config:icinga2]
host = 'https://monitoringserver'
port = 5665
username = 'mqttwarn'
password = ''
cacert = '/opt/service/mqttwarn/icinga.crt'
targets = {
    'vzlogger-q3m' : [ 'busproxy', 'vzlogger-q3m', 'mqttwarn' ],
    'vzlogger-mt174' : [ 'busproxy', 'vzlogger-mt174', 'mqttwarn' ],
    ...
  }

; special config for 'failover' events
[failover]
targets = log:error


; ------------------------------------------
;              Periodic tasks
; ------------------------------------------

[cron]


; ------------------------------------------
;              Icinga monitoring
; ------------------------------------------

[home/devices/vzlogger/#]
format = icinga2_timestamp()
targets = {
    'home/devices/vzlogger/chn0/#': [ 'log:warn', 'icinga2:vzlogger-q3m' ],
    'home/devices/vzlogger/chn1/#': [ 'log:warn' ],
    'home/devices/vzlogger/chn2/#': [ 'log:warn', 'icinga2:vzlogger-mt174' ],
    'home/devices/vzlogger/chn3/#': [ 'log:warn' ],
  }

...

functions.py

import json

from datetime import datetime

def icinga2_timestamp(data, srv):
    icinga2_payload = {
        'exit_status'  : 0,
        'plugin_output': "OK: @ {}".format(datetime.today().ctime()),
        }

    return json.dumps(icinga2_payload)

def icinga2_format(data, srv):
    icinga2_payload = {
        'exit_status'  : 0,
        'plugin_output': "OK: {}".format(data['payload']),
        }

    return json.dumps(icinga2_payload)

[jpmens]

It is wrong.

[jpmens]

Thank you for the report and sorry you’re experiencing trouble. This is going to be a tough one to find because I know many of us are running mqttwarn for months on end without reboots and I’ve not heard of it chewing RAM like this.

My first thought is it must be the icinga plugin, but that’s Python requests only so not super likely. Your configuration is not complex so I cannot really ask you to start slimming it down as there is not much to remove. 😃

Which Python version are you running and which distro? Are there any outstanding updates (particulary to Python) you can apply?