Incorrect error message in evaluation of wrong perf_pattern

MWE.zip

Silly issue with error reporting on wrong code.

       pref_regex = r'Hello\s(\S+)'
       times = sn.extractall(pref_regex, self.stdout, 1, int)
       self.perf_patterns = {
               'Total Time': sum(times)/len(times)
       }

(I know now that I should be using sn.avg(times) but thats not the point here.)

Reframe fails with:

$ reframe -r -c hello.py --performance-report
....
[  FAILED  ] Ran 1/1 test case(s) from 1 check(s) (1 failure(s), 0 skipped)
[==========] Finished on Tue Jan 18 08:49:55 2022 

==============================================================================
SUMMARY OF FAILURES
------------------------------------------------------------------------------
FAILURE INFO for HelloTest 
  * Test Description: HelloTest
  * System partition: PACU-login2:default
  * Environment: builtin
  * Stage directory: /home/ollper01/tmp/rframe/stage/PACU-login2/default/builtin/HelloTest
  * Node list: pacu01
  * Job type: local (id=3696722)
  * Dependencies (conceptual): []
  * Dependencies (actual): []
  * Maintainers: []
  * Failing phase: sanity
  * Rerun with '-n HelloTest -p builtin --system PACU-login2:default -r'
  * Reason: sanity error: rfm_HelloTest_job.out: No such file or directory
------------------------------------------------------------------------------

So it is reporting that the output file doesn’t exist. Which clearly it does, and is not the issue at hand.

Running in a more verbose mode:

$ reframe -vvv -r -c hello.py --performance-report
....
==============================================================================
SUMMARY OF FAILURES
------------------------------------------------------------------------------
FAILURE INFO for HelloTest 
  * Test Description: HelloTest
  * System partition: PACU-login2:default
  * Environment: builtin
  * Stage directory: /home/ollper01/tmp/rframe/stage/PACU-login2/default/builtin/HelloTest
  * Node list: pacu01
  * Job type: local (id=3697015)
  * Dependencies (conceptual): []
  * Dependencies (actual): []
  * Maintainers: []
  * Failing phase: sanity
  * Rerun with '-n HelloTest -p builtin --system PACU-login2:default -r'
  * Reason: sanity error: rfm_HelloTest_job.out: No such file or directory
Traceback (most recent call last):
  File "/home/ollper01/Foundry/reframe/reframe/utility/sanity.py", line 37, in _open
    with open(filename, *args, **kwargs) as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'rfm_HelloTest_job.out'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ollper01/Foundry/reframe/reframe/frontend/executors/__init__.py", line 267, in _safe_call
    return fn(*args, **kwargs)
  File "/home/ollper01/Foundry/reframe/reframe/core/hooks.py", line 78, in _fn
    getattr(obj, h.__name__)()
  File "/home/ollper01/Foundry/reframe/reframe/core/hooks.py", line 32, in _fn
    func(*args, **kwargs)
  File "/home/ollper01/tmp/rframe/hello.py", line 31, in set_sanity_patterns
    'Total Time': sum(times)/len(times)
  File "/home/ollper01/Foundry/reframe/reframe/core/deferrable.py", line 94, in __iter__
    return iter(self.evaluate())
  File "/home/ollper01/Foundry/reframe/reframe/core/deferrable.py", line 69, in evaluate
    ret = self._fn(*fn_args, **fn_kwargs)
  File "/home/ollper01/Foundry/reframe/reframe/utility/sanity.py", line 795, in extractall
    for x in extractiter(patt, filename, tag, conv, encoding))
  File "/home/ollper01/Foundry/reframe/reframe/utility/sanity.py", line 794, in <genexpr>
    return list(evaluate(x)
  File "/home/ollper01/Foundry/reframe/reframe/utility/sanity.py", line 735, in extractiter
    with _open(filename, 'rt', encoding=encoding) as fp:
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/ollper01/Foundry/reframe/reframe/utility/sanity.py", line 41, in _open
    raise SanityError(f'{filename}: {e.strerror}')
reframe.core.exceptions.SanityError: rfm_HelloTest_job.out: No such file or directory

------------------------------------------------------------------------------

Here we can finally spot the actual issue:

File "/home/ollper01/tmp/rframe/hello.py", line 31, in set_sanity_patterns
    'Total Time': sum(times)/len(times)

However, the No such file or directory error is still presented as the issue a number of times, which confused me when debugging this in the real case.

Again, this is low priority, and is a user error - but error messages and debugging are really important when developing test files. No big deal, but hope this MWE can be useful.

[mahendrapaipuri]

Hello @OliverPerks, I think the error message makes sense. The thing is perf_patterns must be deferred expressions and they are “evaluated” after job is finished in post run hooks. Your expression sum(times)/len(times) will trigger the evaluation of the deferred expression even before the job is submitted to the scheduler. At this point, there is no stdout file in the stage directory and that is what ReFrame is complaining about.

[OliverPerks]

Hey Vasileios, Thanks so much for taking the time to explain this - I think I understand now. It’s clearly very nuanced, so I really appreciate the effort into the explanation.

I certainly think having the full path in the error message would have made it much clearer what was going on, so I would really appreciate that (and any more info) that can be provided into the error message.

Again, thanks so much, Olly