performance report does not accept str

Running this check src works:

PERFORMANCE REPORT
------------------------------------------------------------------------------
ExampleRunOnlyTest
- dom:login
   - PrgEnv-cray
      * num_tasks: 1
      * elapsed_time: 8 s
      * filename: 9.c <---------- yes

but fails with:

FAILURE INFO for ExampleRunOnlyTest
  * System partition: dom:login
  * Environment: PrgEnv-cray
  * Stage directory: /apps/common/UES/sandbox/jgp/hpctools.git/reframechecks/common/sphexa/DEBUGJG/JIRA1/stage/dom/login/PrgEnv-cray/ExampleRunOnlyTest
  * Node list: dom101
  * Job type: local (id=4458)
  * Maintainers: ['x']
  * Failing phase: performance
  * Reason: unexpected error: '>=' not supported between instances of 'str' and 'float'
Traceback (most recent call last):
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/frontend/executors/__init__.py", line 163, in _safe_call
    return fn(*args, **kwargs)
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/core/pipeline.py", line 96, in _fn
    func(obj, *args, **kwargs)
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/core/pipeline.py", line 1235, in performance
    self.check_performance()
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/core/pipeline.py", line 1319, in check_performance
    'expected {1} (l={2}, u={3})' % tag))
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/utility/sanity.py", line 727, in evaluate
    return expr.evaluate()
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/core/deferrable.py", line 53, in evaluate
    ret = self._fn(*fn_args, **fn_kwargs)
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/utility/sanity.py", line 546, in assert_reference
    evaluate(assert_bounded(val, lower, upper))
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/utility/sanity.py", line 727, in evaluate
    return expr.evaluate()
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/core/deferrable.py", line 53, in evaluate
    ret = self._fn(*fn_args, **fn_kwargs)
  File "/apps/common/UES/sandbox/jgp/reframe.git.jgphpc/reframe/utility/sanity.py", line 492, in assert_bounded
    if val >= lower and val <= upper:
TypeError: '>=' not supported between instances of 'str' and 'float'

Would be nice to have reframe support this. Would be super nice to have this (being able to use a string for the unit) supported:

      * elapsed_time: 8 (9.c)

[vkarak]

@jgphpc Do not close it, the fact that you got this “crash” is a bug, even if you used it wrongly. It should have given a more user friendly error!

[jgphpc]

You’re right, actually this works without error:

        self.perf_patterns = {
            'elapsed_time': self.elapsed,
            'filename': self.file_line,
        }
        self.reference = {
            '*': {
                'elapsed_time': (0, None, None, 's'),
                'filename': (0, None, None, self.file_name),
            }