feature request: notification of test maintainer on failure

We would like to have optional notifications sent to the maintainer of a test when the test fails.

@rfm.simple_test
class mytest(rfm.RegressionTest):
    def __init__(self):
        super().__init__()
        self.descr = "A Simple test"
        self.valid_systems = ['cori:knl',
                              'gerty:knl']
        self.valid_prog_environs = ['PrgEnv-intel']
        self.maintainers = ['[e-mail]']
        self.notify_maintainers = True

or maybe

self.maintainers = [{'email': '[e-mail]', 'notify' : True}, '[e-mail]']

where non-dict items are treated as "notify" : False.

Is the logging framework the right place to develop something like this? e.g.

logging_config = {
    'level': 'DEBUG',
    'handlers': [
        {
            'type': 'email',
            'level': 'ERROR',
            'format': '[%(asctime)s] %(levelname)s: '
                      '%(check_info)s: %(message)s',
        }
    ]
}

[vkarak]

Hi @brandongc

I think we also want to make sure this is controllable, maybe via a CLI opt-in option like --notifications=on? I don’t want to get spammed by people using reframe to test their work on our development systems.

That’s pretty straightforward now in ReFrame 3.0. We can combine command line options with configuration parameters and environment variables very easily, so such a feature could easily enabled via different ways.

Another thought is to keep it extensible beyond email. One possibility is to also consider webhooks to enable things like Slack integration.

That’s a valid thing to take into account. Being able to open JIRA tickets directly for example would also be nice as a future extension.

And finally I wonder what context could/should be included in a notification beyond “$test failed”.

Regarding this, we’re working in #1377 in generating a json report for the full run (and as a result for each test). This contains all test-related information, so that could be sent as well and perhaps create a summary of it as the notification body. In fact, when this is merged (hopefully for 3.1), you could have any custom external tool reading that json and sending out notifications at will.

[vkarak]

Oh, and most probably this feature request will be postponed to 3.2 due to lack of bandwidth currently 😄