Respect the max age cache control for simple API responses, with option to modify

[notatallshaw]

In #5791 caching simple API responses was turned off, even if a max age cache control response was sent back, this solved one use case at the performance cost of everyone.

When doing multiple pip commands, especially ones that don't require downloading distributions, like pip lock, pip install --dry-run, and pip install where the distributions are already in cache, this contributes to a significant amount of the resolution time.

uv respects the the max age cache control and gives users a way to set their own values for the API and files: https://docs.astral.sh/uv/concepts/indexes/#customizing-cache-control-headers.

I would like to agree in principle we should do the same, and then we can agree on how to do it.

[pfmoore]

I'm happy to agree that in principle. We'll need a different UI to the one uv uses (I'm -1 on reading pip config from pyproject.toml) but we can decide on a UI later, once someone starts work on the feature.

[notatallshaw]

Agreed, this should be a normal option(s), pip should only read from a pyproject.toml when the spec explicitly says it should, like dependency groups (or if #13003 is ever agreed upon and implemented).

I'm not working on this right now, so there may be issues I've not considered, but my general idea was something like --cache-control-api <str> --cache-control-files <str>, with examples of supporting the use case of uploading a new package and then installing it without having to wait for the cache to expire.

We don't currently have any semantics for per-index options, I didn't intend for this feature to solve that, I think that needs to be solved independently.