Is is possible to "parametrize" a benchmark?

I want to benchmark different JSON engines serialization/deserialization functions, with different sets of data. More specifically, I’m trying to convert an already existing set of benchmarks to pytest-benchmark.

Here the contenders is a list of tuples (name, serialization_func, deserialization_func):

@pytest.mark.benchmark(group='serialize default_data')
@pytest.mark.parametrize('serializer',
                         [c[1] for c in contenders],
                         ids=[c[0] for c in contenders])
def test_serializer_benchmark(serializer, benchmark):
    benchmark(serializer, default_data)

@pytest.mark.benchmark(group='deserialize default_data')
@pytest.mark.parametrize('serializer,deserializer',
                         [(c[1], c[2]) for c in contenders],
                         ids=[c[0] for c in contenders])
def test_deserialization_benchmark(serializer, deserializer, benchmark):
    data = serializer(default_data)
    benchmark(deserializer, data)

This will produce two distinct benchmarks tables, one for the serialization function and one for its counterpart. I can go down the boring way of repeating that pattern for each dataset…

What I’d like to achieve is to factorize that to something like the following (that does not work):

@pytest.mark.parametrize('name,data', [('default data', default_data)])
def test_gen(name, data):
    @pytest.mark.benchmark(group=name + ': serialize')
    @pytest.mark.parametrize('serializer',
                             [c[1] for c in contenders],
                             ids=[c[0] for c in contenders])
    def serializer_benchmark(serializer, benchmark):
        benchmark(serializer, data)

    @pytest.mark.benchmark(group=name + ': deserialize')
    @pytest.mark.parametrize('serializer,deserializer',
                             [(c[1], c[2]) for c in contenders],
                             ids=[c[0] for c in contenders])
    def deserializer_benchmark(serializer, deserializer, benchmark):
        serialized_data = serializer(data)
        benchmark(deserializer, serialized_data)

    yield serializer_benchmark
    yield deserializer_benchmark

That way I could reuse the very same code to create benchmarks against all other sets of data, without repeating the code, simply adding them to the initial parametrize:

@pytest.mark.parametrize('name,data', [('default data', default_data),
                                       ('array 256 doubles', doubles),
                                       ('array 256 unicode', unicode_strings),
                                      ])
def test_gen(name, data):
...

Is there any trick I’m missing?

[ionelmc]

It didn’t initially occur to me but you can also do this:

@pytest.mark_parametrize('foo', [1,2,3])
def test_perf(benchmark, foo):
   benchmark.group = '%s - perf' % foo
   benchmark(....)

[ionelmc]

Also, you may set the group from a fixture to reduce boilerplate, eg:

@pytest.fixture(params=[1,2,3])
def foo(benchmark, request):
   benchmark.group = '%s - perf' % request.param
   return request.param

def test_perf(benchmark, foo):
   benchmark(....)

The only constraint is that the fixture needs to be function scoped (as benchmark fixture is).