Errors when ReFrame root directory is not first path in `sys.path`

Hello,

The context here is when I use ReFrame CI integration capabilities, I am getting errors in the dependent jobs. When we use --restore-session flag for dependent job, ReFrame is detecting modules from filenames here.

I am not sure of the rabbit hole details of why it is being done, but the issue I have noticed is that on that stated line, we implicitly assume sys.path[0] if ReFrame root directory. If it is not, ReFrame will have problems finding the modules. For example, in this failed CI pipeline, I printed the sys.path and we can see that ReFrame root directory is not the first path.

This is the directory structure of my repository:

.
├── apps
├── config
├── docs
├── helloworld
├── modules
├── output
├── perflogs
├── reframe
├── spack
└── topologies

And I usually launch tests from root of the repo using: reframe/bin/reframe -C <config> -c <test> -r

[mahendrapaipuri]

Just to give more info on the issue. I guess the issue comes from here -> https://gitlab.com/ska-telescope/sdp/ska-sdp-benchmark-tests/-/blob/93048cf173e1c5c0b3cafbbdc11c309297c1d5ad/apps/level1/imaging-iotest/reframe_iotest.py#L63 . I do not remember why, back then I was appending the current working directory path to sys.path as pointed out in the line. That is why /builds/ska-telescope/sdp/ska-sdp-benchmark-tests is on the front in sys.path.

And why it failed only in CI? Well, I guess import_module_from_file function is called only when invoking ReFrame with --report-file and/or --restore-session flags, if I remember well. In the child pipeline CI jobs, ReFrame is invoked with those flags -> https://gitlab.com/ska-telescope/sdp/ska-sdp-benchmark-tests/-/jobs/1512502926#L46 . I never used these flags when running tests on HPC systems and that is why I did not face this issue even when ReFrame is not on front in sys.path.

I hope that helps!

[mahendrapaipuri]

Hello @rsarm, thanks for coming back to the issue. Actually, the pipeline failed. Let me try to put things more constructively:

• Thanks for pointing me to ReFrame executable and yes, I agree that ReFrame should be on the front in sys.path. However, I printed the sys.path on one of the failed pipelines -> https://gitlab.com/ska-telescope/sdp/ska-sdp-benchmark-tests/-/jobs/1512352442#L91 and we can see here that ReFrame is not the first one in sys.path. This job is the one that generates child pipeline and it always runs successfully. It is the next job, ie the one that executes the child pipeline, that fails. Please have a look at it -> https://gitlab.com/ska-telescope/sdp/ska-sdp-benchmark-tests/-/jobs/1512502926 .
• I am not quite sure why /builds/ska-telescope/sdp/ska-sdp-benchmark-tests path popped up in there. I have noticed that it happened only in the CI jobs. Maybe it is something GitLab CI specific? I never faced this issue when running the tests on HPC platforms.
• In any case, we can notice from this failed job that we can indeed mess up sys.path which can fail ReFrame. That is why I suggested to check the sys.path in import_module_from_file function to make it more resilient.

Is it more clear now? If not, we can have a quick chat about the issue. Cheers