Incomplete node list

On lumi the command sacct -S <start_tim> -P -j <jobid> -o jobid,state,exitcode,end,nodelist doesn’t always give the complete node list since the beginning of the job. It would print for instance

JobID|State|ExitCode|End|NodeList
1305969|PENDING|0:0|Unknown|nid005828
1305969.batch|RUNNING|0:0|Unknown|nid005828

and a bit later it starts giving the whole information

JobID|State|ExitCode|End|NodeList
1305969|RUNNING|0:0|Unknown|nid[005828-005831]
1305969.batch|RUNNING|0:0|Unknown|nid005828
1305969.0|RUNNING|0:0|Unknown|nid[005829-005831]

This is a problem for reframe because it updates the node list only once https://github.com/reframe-hpc/reframe/blob/a5b66c7c41d7cc884893642fd4d9331b146a3c16/reframe/core/schedulers/slurm.py#L383-L392

[ireed]

Getting the nodelist through job.nodelist does not work consistently, regardless of which stage i try to do it. I do not expect it to work before or during the compile stage. But, for performance or sanity stage (or, any time after the run stage), it is incredibly useful to know which nodes i have and how many. At the moment, i do not know how to reliably get this information through the framework without hacks.

[vkarak]

The problem is that we peek into the nodelist only once, the first time we get back a non-empty node list by sacct or squeue as @rsarm has pointed out. Apparently, Slurm does not fill it up at once, so that’s why we miss it sometimes and sometimes we don’t. I think the best solution is to retrieve the value everytime, but do not issue scontrol to unwrap the node list until the job has finished, so that we issue scontrol once.