Treat QoS as a first-class parameter to support flexible Partition/QoS Slurm configurations

[tbigot]

I'm opening this issue to suggest an improvement regarding the handling of Slurm's Quality of Service (QoS). The current implementation seems to assume that QoS is a secondary parameter, whereas for many cluster configurations, it is as fundamental as the partition itself.

Current State and Limitations

Currently, the plugin's handling of QoS is limited:

1. The primary method for setting a QoS is via slurm_extra. While functional, this treats it as an arbitrary command-line argument rather than a structured resource that the plugin can reason about for its advanced scheduling logic.
2. A --slurm-qos command-line option was recently added (PR feat: new flag for SLURM qos #351), but it appears primarily tied to reservation management and does not integrate natively into rule resources or profiles in the same way as partition or mem-mb.
3. The automatic partition selection feature (PR feat!: proposal for dynamic partition selection #321) is a great step forward, but it is incomplete because it does not consider QoS. However, the maximum allowed walltime is often determined by the partition and/or the QoS, rather than by the partition alone.

Use Case: Flexible Slurm Configurations

In Slurm environments that leverage both partitions and QoS for policy enforcement, like ours, the actual job limits can be complex.

Key scheduling parameters, such as the maximum walltime (MaxWall for a QoS, MaxTime for a partition), preemption rules, or resource limits (GrpTRES), can be defined at the partition level, the QoS level, or both. Consequently, ignoring the QoS can lead to an incorrect assessment of a job's effective limits, often resulting in its rejection by the scheduler.

Concrete example from our cluster:

• The common partition has several associated QoS:

  • qos=long: unlimited walltime
  • qos=normal: 24-hour limit
  • qos=fast: 2-hour limit (default)
  • qos=ultrafast: 5-minute limit

• If I submit a Snakemake rule requiring walltime_min: 180 (3 hours), the plugin must specify not only --partition=common but also --qos=normal.

• If only the partition is specified, the job will inherit the default QoS (fast), and Slurm will immediately reject it for exceeding the 2-hour limit. The current auto-partition selection mechanism would fail here, as it cannot know that by selecting a different QoS, the common partition would become a valid choice.

Proposed Solution

To make the plugin compatible with these more flexible configurations, I would suggest treating QoS as a first-class parameter:

1. Make qos a standard submission parameter:

   • Allow its definition in a profile via a dedicated option, e.g., --slurm-qos "qos_name".
   • Allow its specification directly within a rule's resources: resources: qos="normal".
   • Snakemake would then build the sbatch command with the --qos={resources.qos} flag.

2. Integrate QoS into the resource auto-selection logic:

   • The logic introduced in PR feat!: proposal for dynamic partition selection #321 should be extended to search for a valid (partition, qos) combination that satisfies the rule's requirements.
   • This would involve querying Slurm for the limits associated not just with partitions, but also with the QoS allowed on those partitions.

Thank you for your consideration.

[cmeesters]

Hi,

thanks for bringing this to our attention.

... for many cluster configurations ...

There are indeed two different philosophies in how to achieve the same scheduling result. Whereas I think the one favouring partitions with different time or resources is prevalent, I do indeed realize that having one partition for a physical resource with different QoSes for the non-physical onces, e.g. walltime, is meanwhile favoured by some admins.

As for the proposed solutions: Did you know that with Snakemake you can use every CLI option as a resource definition (hard-coded in your Snakefile(s) (not recommended) and a workflow profile or profile), too? So, in essence, suggestion No. 1 ought to be working. Except, that we choose to prefix with slurm_ to different between this and potential other executors.

As for suggestion 2: We will implement this. I am, however, not sure what the best way would be. Perhaps something like:

partition_name:
    runtime: "5m, 2h, 24h"
    slurm_qos: "ultrafast, fast, normal"

That would be easy to write, but error-prone. Any other option, I am thinking about, is more tedious.

Please understand, it might take some time - "my" cluster is currently offline for maintenance, and I'm busy.

[tbigot]

Hi @cmeesters, and thanks for your fast response!

First, I totally understand that you are busy. This isn’t urgent. I had naively assumed qos was a standard feature for all Slurm setups, and so that it would be implemented one day or the other in the plugin. But I realize now that if users with configurations like ours don't bring them to your attention, there's no way for you to anticipate the need. 🙂

Regarding Suggestion 2, what if we defined requirements for specific (partition, qos) pairs?. For instance:

common,ultrafast:
    runtime: "5m"

Regarding Suggestion 1, I was not aware that it was possible to invoke slurm_qos, I will test it very soon.