Override adaptive scaling of Dask/Coiled cluster #847
Description
One of the easiest ways to deploy Cubed today is to run on a Dask cluster or on Coiled functions. A big downside of this approach is that autoscaling is delegated to dask/coiled, which uses heuristics that won't be anywhere near optimal for the "perfectly horizontal" bursty workloads that Cubed creates.
However @samantha-earthmover tells me that it's actually possible to override the auto-scaling behaviour of dask (and hence Coiled too, IIUC?).
https://docs.dask.org/en/latest/adaptive.html#scaling-heuristics
I'm not sure of the exact details of how best to do this yet, but if we e.g. used cubed callbacks to tell dask to scale up and down to the exact optimal size at every stage, we could right-size the cluster at all points in the computation. This would be a huge boon: dask-based computations would run faster and avoid wasting resources.