`self.perf_patterns` as a dictionary

Hi, I would like to run the osu-micro-benchmark from size 1 to 4194304 in a single run.

To get the performance metric I am using:

    @run_before("performance")
    def set_perf_patterns(self):
        regex = r"^(?P<size>\d+)\s+(?P<perf>\S+)"
        sizes = sn.extractall(regex, self.stdout, "size", int)
        perfs = sn.extractall(regex, self.stdout, "perf", float)
        self.perf_patterns = dict(zip(sizes, perfs))

But reFrame errors as the dictionary keys are not hashable. Is there a better way to defer this operation?

Thanks

[jjotero]

Hi @yellowhat ! I can see a few things that could help here.

1. Hooks do not run in the test’s stage directory (the reason is that one could have a pre-setup hook, which runs before the stage directory has even been created). You have to manually change the directory. You can do that with the osext module under utility.
2. The sn.extractall builds a deferrable expression (i.e. it captures the arguments passed to the function and it does not evaluate the actual function until you call sn.evaluate on the returning object). The return object is not hashable, so you can’t use it as keys in any kind of dictionary.

@run_before('performance')
def set_perf_patterns(self):
    with osext.change_dir(self.stagedir):
        ...
        sizes = sn.evaluate(sn.extractall(regex, self.stdout, "size", int))
        ...

[jjotero]

Yup, the keys of perf_patterns must be strings 👇 https://github.com/eth-cscs/reframe/blob/152d697d59250f30664e7c7502ebf3680fd8af37/reframe/core/pipeline.py#L645