@@ -249,6 +249,8 @@ Rust code being written today.
249249
250250# Benchmark update policy
251251
252+ ## Background
253+
252254rustc-perf is a "living benchmark suite" that is regularly changed. Some
253255benchmarks in rustc-perf are verbatim copies of third-party crates. We
254256periodically do a mass update of these benchmarks.
@@ -265,26 +267,19 @@ Costs of this approach:
265267 and they provide the greatest continuity.
266268- If the code hasn't changed much, it won't have much effect.
267269
268- To balance these requires choosing a good refresh period. A period of three
269- years is reasonable:
270- - It's not too frequent (as 1 year would be, for example) while avoiding some
271- sub-optimal occurrences from the past (e.g. in 2022 numerous crates that were
272- more than five years old were updated).
273- - It aligns with the Rust edition cycle.
270+ ## Update policy
274271
275- Update policy
276- - The third-party crates should be updated in the year following a new edition.
277- (Based on the current edition cadence, this is every three years. If the
278- edition cadence changes in the future, this period should be reconsidered.)
279- The update should occur roughly six months after the edition hits a stable
280- Rust release, to give crate authors a decent amount of time to update to the
281- new edition.
272+ - The third-party crates should be updated every three years. This is a
273+ reasonable refresh period that is neither too short or too long. It happens
274+ to match the Rust edition cycle, but this is just coincidence.
282275- All third-party crates that have had at least one new release should be
283276 updated, even if not much code has changed. This avoids having to make
284277 decisions about whether a crate has changed enough.
285278- When doing this mass update, there may be some benchmarks that are deemed no
286- longer interesting and removed. Likewise, there may be some new benchmarks
287- that are added.
279+ longer interesting and removed. For example, in the 2022 update we found that
280+ the `futures` crate was no longer interesting because all the functionality
281+ had been split into sub-crates that rustc-perf doesn't measure. Likewise,
282+ there may be some new benchmarks that are added.
288283- New versions should be added before old versions are removed, to ensure
289284 continuity of profiling coverage.
290285- The ad hoc addition and removal of individual benchmarks can continue
0 commit comments