By using GitHub-hosted runners for compilation, we may get better overall throughput and shorter wait times for developers when a bunch of PRs are queued.
Another potential advantage here is that the setup of toolchains will only be required in the compiling action runner, hopefully making the benchmarking runners just "vanilla" installs of their respective OSes and therefore easier to spin up.
We should confirm whether this fits in our free allowance provided to private projects (2,000 min/month, 500MB of storage for artifacts).
We will also need to keep the "single Python command to do everything" refactor working for non-Github-Action-based environments.
Credit for the idea goes to @brandtbucher.