Abstract reframe tests

Intro

In its current version, the base definition of a reframe test is typically in the same file or directory as its derived tests. From there on, test specialisation is done through a mix of class inheritance and if statements. This implies that tests are rather difficult to share across different users, who might be using a completely different system, programming environment, compiler options, etc.

The proposed feature of abstract reframe tests, enables abstracting away individual reframe tests in simple python modules, where test specialisation occurs by using the given python modules and extending the desired reframe test through simple class inheritance. Similarly to C++ abstract classes, abstract reframe tests have members that are not implemented (e.g. the list of valid partitions, list of valid programming environments, modules, etc) and these abstract members are implemented by a child class. Therefore, the use of abstract tests would allow the reframe community to contribute their tests into a shared test library, where every single test in this library is not specialised to any particular system or programming environment.

A Hello World Example

Because these abstract cases might not be complete (i.e. they might have abstract members), the tests cannot be registered by reframe by using the decorators @rfm.simple_test and @rfm.parameterized_test. Instead, the new proposed decorators @rfm.simple_abstract_test and @rfm.parameterized_abstract_test are used to flag a class as an abstract test. The @rfm.simple_abstract_test decorator simply enables the child tests to keep track of the directory where the abstract class was defined. This means that these child tests will be able to retrieve the source of the abstract test correctly (see example below). For more complex cases, the decorator @rfm.parameterized_abstract_test enables setting default test parameters that can be later overridden by any child test. Also, for a more advanced use, multiple abstract tests can be chained together, enabling a complex combination of test parameters. To illustrate all this, we show a simple hello world example, where the source

#include <iostream>
  
#ifndef EXTRA
# define EXTRA "base"
#endif
int main()
{
  std::cout << "Hello, World! (" <<  EXTRA << ")" << std::endl;
  return 0;
}

takes a preprocessor flag EXTRA that extends the most basic Hello, World! output. This file is located under the directory where the reframe abstract tests are defined; for example, in $RFM_DIR/abstractTestLibrary/hello/src/hello.cpp. Thus, the abstract reframe tests would live in $RFM_DIR/abstractTestLibrary/hello/hello.py.

import reframe as rfm
import reframe.utility.sanity as sn

# Log the abstract test and do not register the test.
@rfm.simple_abstract_test
class HelloAbstract(rfm.RegressionTest):
    def __init__(self):
        self.valid_systems = ['*']
        self.valid_prog_environs = ['*']
        self.sourcepath = 'hello.cpp'
        self.sanity_patterns = sn.assert_found(r'Hello, World\!', self.stdout)


# Log the abstract test and do not register the test.
# Note that we're extending a simple_abstract_test to a parametrized_abstract_test!
@rfm.parameterized_abstract_test(['a'], ['b'], ['c'])
class HelloParameterizedAbstract(HelloAbstract):
    def __init__(self, flags):
        super().__init__()
        self.build_system = 'SingleSource'
        self.executable = 'a.out'
        self.build_system.cxxflags = [r'-DEXTRA="\"%s\""' % flags]
        self.sanity_patterns = sn.assert_found(r'Hello, World\!', self.stdout)

Note how these tests are completely system agnostic. In fact, due to their simplicity, these classes are not even abstract classes and are complete and ready to run as reframe tests. However, these tests could be made fully abstract by removing self.valid_systems and self.valid_prog_environs from the HelloAbstract class. In this example, the class HelloAbstract is a simple abstract case, and the class HelloParameterizedAbstract is a parametrised abstract case with default parameters ['a'], ['b'], ['c']. These default parameters can be later overridden, extended, filtered, etc. but we’ll get to that later.

Simple abstract test

So, we now have our abstract test library, but how do we use that? Easy, we simply inherit the abstract test from our python module containing the abstract tests and wrap it with @rfm.simple_test.

import reframe as rfm
import reframe.utility.sanity as sn

# Import the python module with the abstract tests.
import abstractTests.hello.hello_abstract_tests as tt 


@rfm.simple_test
class HelloTest(tt.HelloAbstract):
    '''
    Inherit the simple_template and register it as a test.
    This test will use identical settings to those defined in the template.

    Logically, the class tt.HelloAbstract is the HelloAbstract class defined in
    the code snippet above.
    '''

    def __init__(self):
        super().__init__()

        # If the class tt.HelloAbstract would not have (for example) the test 
        # parameters self.valid_systems or self.valid_prog_environs, they 
        # would have to be defined in here.

Therefore, we now just have to simply point reframe to this file in order to run the HelloTest check. Note how the cpp source code is not in the same directory as this HelloTest class, but because the class tt.HelloAbstract was decorated as a simple abstract test, reframe is able to retrieve the sources from the right directory.

Parametrised abstract test

Now let’s have a look into the more advanced features using @rfm.parameterized_abstract_test. Remember how the parametrised abstract test in our abstract test library had some default parameters defined? To use those default parameters we just need:

import reframe as rfm
import reframe.utility.sanity as sn
import abstractTests.hello.hello_abstract_tests as tt

@rfm.simple_test
class HelloDefaultParameters(tt.HelloParameterizedAbstract):
    '''
    Inherit from a parametrized abstract test and run the case with the
    parameters defined in the abstract test definition. Note that __init__
    must have the same arguments than the class it inherits from. This is
    because the simple_test decorator takes care of feeding this class the
    default test arguments defined on the parametrized abstract test
    tt.HelloParameterizedAbstract class.
    ''' 
        
    def __init__(self, flags):
        super().__init__(flags)

It cannot get easier than this. The decorator @rfm.simple_test is aware that the inherited class is a parametrised abstract test, and it does take care of feeding this child class the right test parameters. Sure, but what if you don’t want those default parameters? Then you simply override them 😃

# Same imports here as in the snippet above.

@rfm.parameterized_test(['d'])
class HelloOverrideParameters(tt.HelloParameterizedAbstract):
    def __init__(self, flags):
        super().__init__(flags)

Here, @rfm.parameterized_test is aware that the class tt.HelloParameterizedAbstract is in fact an parametrised abstract test, and it overrides the default parameters ['a'], ['b'], ['c'] and replaces them by ['d']. So what if we now want to combine all the parameters?

@rfm.parameterized_test(['d'], inherit_params=True)
class HelloExtendParameters(tt.HelloParameterizedAbstract):
    '''
    Now we extend the default parameter list.
    '''

    def __init__(self, flags):
        super().__init__(flags)

The decorator argument inherit_params controls whether the test parameters are overridden or not. Now, what if we just want some of the existing parameters and not all of them?

@rfm.parameterized_test(inherit_params=True, filt_params=lambda x: x[1:])
class HelloFilterParameters(tt.HelloParameterizedAbstract):
    '''
    Here we just filter out some of the default test parameters.
    '''

    def __init__(self, flags):
        super().__init__(flags)

The argument filt_params takes a function that modifies the argument tuple. In this example, we simply pop the first of the test parameters ['a']. Of course, we can also extend the parameters AND filter some of the existing ones out:

@rfm.parameterized_test(['d'], inherit_params=True, filt_params=lambda x: x[1:])
class HelloFilterExtendParameters(tt.HelloParameterizedAbstract):
    ''' 
    Now we filter some of the default test parameters out and also extend the
    list with new parameters.
    '''

    def __init__(self, flags):
        super().__init__(flags)

And lastly, to make even more complex combinations of the test parameters, fuse_params takes any user defined function to combine the existing test parameters with the new ones. For example, here we transform the argument list from ['a'], ['b'], ['c'] to ['ad'], ['bd'], ['cd']:

def expand(x, y):
    '''This function could be made more elegant with itertools, but it's just here illustrate the point.'''
    temp = []
    for i in x:
        for j in y:
            temp.append([i[0]+j[0]])
    return tuple(temp)


@rfm.parameterized_test(['d'], inherit_params=True, fuse_params=expand)
class HelloFuseParameters(tt.HelloParameterizedAbstract):
    '''
    Lastly, we introduce some custom combination of the existing test
    parameters and the new ones.
    '''

    def __init__(self, flags):
        super().__init__(flags)

As a last side note, it is also worth mentioning that an abstract test can be created by inheriting from other abstract tests. This allows the user to potentially use the filtering and fusing methods over multiple steps to write complex test argument lists in a very compact way. Creating an abstract test by inheriting from another abstract test will not override the base source directory as long as the decorator’s argument set_base_path is NOT set to True (defaults to False).

@rfm.parameterized_abstract_test(['d'], inherit_params=True)
class HelloAbstractExtendedParameters(tt.HelloParameterizedAbstract):
    '''
    This does NOT register the test - it is still an abstract test.
    Here we just create an abstract test from another abstract test.
    '''

    def __init__(self, flags):
        super().__init__(flags)
# and


@rfm.simple_test
class HelloExtendParameters2(HelloAbstractExtendedParameters):
    '''
    But this DOES registers the tests - no longer an abstract test.
    '''

    def __init__(self, flags):
        super().__init__(flags)

Implementation

The bulk of the implementation to the above functionality takes place in reframe/core/decorators.py, where the abstract test decorators look as follows

def simple_abstract_test(cls, set_base_path=False):
    '''
    Class decorator for injecting the _rfm_abs_test_prefix class attribute.
    This attribute is used on the reframe pipeline to set the default dirs
    of the check.

    The decorated class must derive from
    :class:`reframe.core.pipeline.RegressionTest`.  This decorator is also
    available directly under the :mod:`reframe` module.

    :arg set_base_path: Flag to control the base test path override. If
         an abstract test inherits from another abstract test, the base
         test path will remain as the base abstract case's one, unless
         set_base_path is se to True.

    .. note::
    This function does not register the reframe test.
    '''
    if not hasattr(cls, '_rfm_abs_test_prefix') or set_base_path:
        cls._rfm_abs_test_prefix = os.path.abspath(
            os.path.dirname(inspect.getfile(cls)))
    return cls


def _set_filt_params(arg):
    return (lambda x: x) if (arg is None) else arg


def _set_fuse_params(arg):
    return (lambda x, y: x+y) if (arg is None) else arg


def parameterized_abstract_test(*inst, inherit_params=False, filt_params=None,
                                fuse_params=None, set_base_path=False):
    '''
    Class decorator extends the simple_abstract_test decorator and injects
    the _rfm_abs_test_params class attribute. This attribute holds the args to
    be used at a later stage during the registration of the checks as 
    parametrized tests (*inst). See below parameterized_test for a full 
    description of the arguments.

    The decorated class must derive from
    :class:`reframe.core.pipeline.RegressionTest`.  This decorator is also
    available directly under the :mod:`reframe` module.

    .. note::
    This function does not register the reframe test.
    '''

    _filt_params = _set_filt_params(filt_params)
    _fuse_params = _set_fuse_params(fuse_params)

    def _do_not_register(cls):
        cls._rfm_abs_test_params = inst if (
            not hasattr(cls, '_rfm_abs_test_params') or
            not inherit_params) else _fuse_params(
            _filt_params(cls._rfm_abs_test_params), inst)
        return simple_abstract_test(cls, set_base_path)

    return _do_not_register

Also, in order to extend the existing @rfm.simple_test and @rfm.parameterized_test to deal with these abstract tests as described above, these decorators have to be modified as

def simple_test(cls):
    '''Class decorator for registering parameterless tests with ReFrame.

    The decorated class must derive from
    :class:`reframe.core.pipeline.RegressionTest`.  This decorator is also
    available directly under the :mod:`reframe` module.

    If the decorated class inherits from a parametrized abstract test, this
    function will register all the default parametrized tests defined on the
    parent abstract test class.

    .. versionadded:: 2.13
    '''

    _validate_test(cls)
    if hasattr(cls, '_rfm_abs_test_params'):
        for args in cls._rfm_abs_test_params:
            _register_test(cls, args)

    else:
        _register_test(cls)

    return cls


def parameterized_test(*inst, inherit_params=False,
                       filt_params=None, fuse_params=None):
    '''Class decorator for registering multiple instantiations of a test class.

    The decorated class must derive from
    :class:`reframe.core.pipeline.RegressionTest`. This decorator is also
    available directly under the :mod:`reframe` module.

    :arg inst: The different instantiations of the test. Each instantiation
         argument may be either a sequence or a mapping.
    :arg inherit_params: If False, the test will discard any potential default
         parameters inferited from a parameterized_abstract_test. Contrarily, if
         set to True, *inst will add to the parameters under cls._rfm_abs_test_params.
    :arg filt_params: Filtering function to eliminate/expand/manipulate the existing
         test parameters inherited from a parameterized_abstract_test. It only has an
         effect if the argument inherit_params is True. By default, it leaves the
         existing test parameters unaltered.
    :arg fuse_params: Combinatorial function that allows for any complex merging
         operation of the two sets of test parameters cls._rfm_abs_test_params and inst.
         By default, it simply joins both sets.

    .. versionadded:: 2.13

    .. note::
       This decorator does not instantiate any test.  It only registers them.
       The actual instantiation happens during the loading phase of the test.
    '''

    _filt_params = _set_filt_params(filt_params)
    _fuse_params = _set_fuse_params(fuse_params)

    def _do_register(cls):
        _validate_test(cls)
        abs_test_args = _filt_params(cls._rfm_abs_test_params) if (hasattr(
            cls, '_rfm_abs_test_params') and inherit_params) else ()
        for args in _fuse_params(inst, abs_test_args):
            _register_test(cls, args)

        return cls

    return _do_register

[jjotero]

Right, I think we should break things up a little bit more. How about introducing the concept of base tests? In this context, and similarly to c++, a base test would be a non-specialised test (not necessarily an abstract test) that lives in a library and might have the test sources with it. This would mean that tests that inherit from this base test will need to retrieve the sources (if any) from the base test’s directory. This would replace the functionality introduced by the simple_abstract_test decorator shown above, which now that I think about it, the name was kinda wrong. To get this done, we just modify the metaclass slightly in order to insert the cls._rfm_base_test_prefix attribute. This would be then picked up by the reframe pipeline and that’s job done. See example below.

import os, inspect

class TestMeta(type):
    def __new__(cls, name, bases, namespace, **kwargs):
        return super().__new__(cls, name, bases, namespace)

    def __init__(cls, name, bases, namespace, base_test=False, **kwargs):
        super().__init__(name, bases, namespace, **kwargs)
        if base_test:
            cls._rfm_base_test_prefix = os.path.abspath(
                os.path.dirname(inspect.getfile(cls)))


class BaseTest(metaclass=TestMeta, base_test=True):
    '''
    Test where the sources are. It sets the cls._rfm_base_test_prefix attribute.
    This goes in the test library and may or may not be an abstract test.
    '''
    def __init__(self):
        pass

    
class DerivedTest(BaseTest):
    '''
    This derived test could be in a different file from BaseTest. 
    Because this derived test inherits from BaseTest, reframe will
    retrieve the right sources (located in the same dir as BaseTest)
    using the cls._rfm_base_test_prefix attribute.
    '''
    pass
    

if __name__ == '__main__':
    foo = DerivedTest()
    try:
      print(foo._rfm_base_test_prefix)
    except:
      pass

Perhaps this can make for a nice and simple PR on its own?

I’ll share what I’ve got on the parameterized abstract tests later 😃

[vkarak]

I will close this issue. I have created individual issues with all the ideas that we came up with here. Have a look at #1567, #1568 and #1569. Please feel free to comment in those issues directly.