Using ReFrame to test an important number of nodes using `--distribute` and avoid massive queing

[MarcoMagl]

ReFrame Version: 4.8.4
Python version: 3.12.3
Scheduler: slurm

What I want to achieve: I want to test all nodes of a node list. I have >20 tests per node and 500 nodes

Problem:

• Some of the nodes I want to test are being allocated already by some users.
• If I launch my tests with --distribute=idle, the nodes already allocated will not be tested
• If I launch my tests with --distribute=avail and I am not lucky, ReFrame might queue a lot of tests on the nodes allocated by other users. At some point, even if system.partitions.max_jobs is high enough, I will reach the limit of job that I am allowed to submit (MaxSubmit in sacctmgr show association). I end up with a lot of jobs queued because ReFrame tried to launch the tests on the nodes that were allocated

squeue -u $USER --state=pending | wc -l
568

but almost no test is running

squeue -u $USER --state=running | wc -l
2

in short: my pipeline of tests is totally stuck even if some nodes are idle! To rephrase that and present the problem from a different perspective: ReFrame submitted the jobs to the nodelist in such a way that it did not prioritize the runs on nodes that were idle. As some nodes were allocated and ReFrame queued jobs on them, I ended up reaching the maximum number of jobs allowed for my slurm account.

In the issues, I did not find a similar problem.
Has anyone an idea how to overcome this issue? That would be really helpful!

[vkarak]

Afaict ReFrame behaves as expected here. The --distribute=avail will submit jobs also to nodes that are currently allocated. This is documented:

Note that if a node is currently allocated to another job it is still considered as “available.”

If you want your ReFrame run to go through immediately you have to use --distribute=idle. ReFrame will not iteratively submit jobs as the nodes in the nodelist become idle. Also the --distribute will submit all the jobs at once.

Overcoming this issue can only be done externally to reframe where in a loop you iteratively invoke reframe with --distribute=idle on the nodes that have not been tested yet in your nodelist.

[MarcoMagl]

Hello @vkarak,

Thanks for your answer.

I know that --distribute=avail will submit some jobs also on the allocated node, but I thought that somehow, there would be a way to first start testing nodes that are idle in order to make the test pipeline progress and then queue the jobs that can't be run right away because nodes are being allocated

[vkarak]

but I thought that somehow, there would be a way to first start testing nodes that are idle in order to make the test pipeline progress and then queue the jobs that can't be run right away because nodes are being allocated

You mean that the first max_jobs to be submitted should include the idle nodes?

[MarcoMagl]

Yes! Don't you think it would be great to use --distribute=avail to reduce the test list in the pipe by running as many jobs as possible on idle nodes, and then queue the jobs (this is inevitable) on the allocated nodes?

In the meantime, I might have a way to dodge this.
Instead of having one reframe process that triggers one single pipeline on all the nodes to tests, I use the local scheduler to launch the tests on the same node where reframe runs.
For example: instead of running a pipeline with 20 tests per nodes on 10 nodes, I launch 10 pipelines (one on each node to test) with 20 tests each

[vkarak]

Yes! Don't you think it would be great to use --distribute=avail to reduce the test list in the pipe by running as many jobs as possible on idle nodes, and then queue the jobs (this is inevitable) on the allocated nodes?

Yes, that makes sense and maybe it's not that difficult to implement; we'd need to sort the nodes based on their state while instantiating the per-node tests. I am converting this to a feature request.

In the meantime, I might have a way to dodge this.
Instead of having one reframe process that triggers one single pipeline on all the nodes to tests, I use the local scheduler to launch the tests on the same node where reframe runs.

Yes, this is a valid workaround too.