diff --git a/_freeze/learn/work/calibration-splits/index/execute-results/html.json b/_freeze/learn/work/calibration-splits/index/execute-results/html.json new file mode 100644 index 00000000..021d8040 --- /dev/null +++ b/_freeze/learn/work/calibration-splits/index/execute-results/html.json @@ -0,0 +1,15 @@ +{ + "hash": "70be4ff5c69007c927cb839ec114601b", + "result": { + "engine": "knitr", + "markdown": "---\ntitle: \"Automatic calibration splits for resamples\"\ncategories:\n - post-processing\n - calibration\n - rsample\ntype: learn-subsection\nweight: 6\ndescription: | \n Learn how tidymodels generates automatic calibration sets for post-processing.\ntoc: true\ntoc-depth: 2\ninclude-after-body: ../../../resources.html\n---\n\n\n\n\n\nWhile preprocessing is the transformation of the predictors prior to a model fit, post-processing is the transformation of the predictions after the model fit. This could be as straightforward as limiting predictions to a certain range of values to as complicated as transforming them based on a separate calibration model. \n\nA calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to accidentially overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed).\n\nrsample provides a collection of functions to make resamples for empirical validation of prediction models. So far, the assumption was that the prediction model is the only model that needs fitting, i.e., a resample consists of an analysis set and an assessment set.\n\nIf we include calibration into our workflow (bundeling preprocessing, (primary) model, and post-processing), we want an analysis set, a calibration set, and an assessment set. \n\nHere we describe how we create these sets automatically when you resample or tune a workflow with calibration. Note that you currently cannot create manual calibration splits and supply them to the functions in tune, like `fit_resamples()` or `tune_grid()`. If you want to do this, leave us a [feature request](https://github.com/tidymodels/rsample/issues/new?template=feature_request.md).\n\n## General principles\n\nFirst principle: we leave the assessment set purely for performance assessment and rather treat the analysis set as \"the data to fit\" - except now we split it to fit two different models instead of just one. Or in other words, we take the data for the calibration set from the analysis set, not from the assessment set. \n\nIf you compare a model with calibration to one without, and you use the same resamples, you are also using the same assessment sets.\n\n\"Taking data from the analysis set\" means splitting up the analysis set to end up with ... an analysis set and a calibration set. Now we have two sets called analysis set, that's confusing. If we need to distinguish them, we'll refer to them as \"outer\" and \"inner\" analysis set for \"before\" and \"after\" the split for a calibration set.\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](images/analysis-calibration-assessment.jpg){fig-align='center' height=350}\n:::\n:::\n\n\nIt is important to note that including calibration requires more data than fitting a workflow without calibration. For small datasets, you might run into situations where fitting either model (primary model or calibration model) is not particularly stable - or you might not even be able to construct a calibration set properly.\n\nSecond principle: we try to mimick the data splitting process used to create the (outer) analysis set and the assessment set.\n\nAnd lastly, the third principle: if we cannot make a calibration split, we fall back onto an empty calibration set rather than erroring.\n\nLet's look at v-fold cross-validation as an example.\n\n\n### Example: v-fold cross-validation\n\nFor v-fold cross-validation, we split the data into `v` folds and create `v` resamples by always using one of the folds as the assessment set while combining the other folds into the analysis set. \n\nSince we want to have a calibration split similar in spirit, we are splitting the (outer) analysis set into (inner) analysis set and calibration set with the same proportion of observations going into the analysis set: 1 - 1/v.\n\nThis tidymodels-internal split of the (outer) analysis set that happens automatically during resampling is implemented in `internal_calibration_split()`. This function has methods for various types of resamples. They are exported to be used by tune but not intended to be user-facing. We are showing them here for illustration purposes.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nset.seed(11)\nr_set <- vfold_cv(warpbreaks, v = 5)\nr_split <- get_rsplit(r_set, 1)\nr_split\n#> \n#> <43/11/54>\n\n# proportion of observations allocated for fitting the model in the outer split\nnrow(analysis(r_split)) / nrow(warpbreaks)\n#> [1] 0.7962963\n\nsplit_cal <- internal_calibration_split(r_split, .get_split_args(r_set))\nsplit_cal\n#> \n#> <34/9/43>\n\n# similar proportion of observations allocated for fitting the model in the inner split\nnrow(analysis(split_cal)) / nrow(analysis(r_split))\n#> [1] 0.7906977\n```\n:::\n\n\nFor most types of resamples, these principles are straightforward to apply. However, for resamples of ordered observations (like time series data) and bootstrap samples, we need to consider additional aspects.\n\n## Splits for ordered observations\n\nrsample's functions for creating resamples of ordered observations are the `sliding_*()` family: `sliding_window()`, `sliding_index()`, and `sliding_period()`.\n\n- `sliding_window()` operates on the ordered rows, allowing us to specify how far to look back to create an analysis set, how far to look ahead to create an assessment set and then sliding that across the data to create a set of resamples.\n- `sliding_index()` works in much the same way, with the key difference being that what we are sliding over aren't the rows directly but rather an index. This is useful when dealing with irregular series of data: while the number of rows which fall into a window can vary, the window length is the same across all resamples.\n- `sliding_period()` takes the idea of the `sliding_index()` function one step further and aggregates the index into periods before sliding across. This is useful for aggregating, e.g., daily data into months and then defining the analysis and assessment set in terms of months rather than the daily index directly.\n\n### Row-based splitting\n\nLet's start with the row-based splitting done by `sliding_window()`. We'll use a very small example dataset. This will make it easier to illustrate how the different subsets of the data are created but note that it is too small for real-world purposes. Let's use a data frame with 11 rows and say we want to use 5 for the analysis set, 3 for the assessment set, and leave a gap of 2 in between those two sets. We can make two such resamples from our data frame.\n\n![](images/calibration-split-window.jpg)\n\nOr, in code:\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\ndf <- data.frame(x = 1:11)\nr_set <- sliding_window(df, lookback = 4, assess_start = 3, assess_stop = 5)\nr_set\n#> # Sliding window resampling \n#> # A tibble: 2 × 2\n#> splits id \n#> \n#> 1 Slice1\n#> 2 Slice2\n```\n:::\n\n\nLet's take that first resample and split it into an (inner) analysis set and a calibration set. \n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 1)\nanalysis(r_split)\n#> x\n#> 1 1\n#> 2 2\n#> 3 3\n#> 4 4\n#> 5 5\n```\n:::\n\n\nTo mimick the split into (outer) analysis and assessment set, we'll calculate the proportion of the analysis set and apply it to the observations available for fitting both the primary model and the calibration model, creating our (inner) analysis set. Since the remaining observations could be for the calibration model or a gap, we also calculate the proportion of the assessment set and use it to construct the calibration set. \n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlookback <- 4\nassess_start <- 3\nassess_stop <- 5\n\n# absolute sizes in the outer split\nlength_window <- lookback + 1 + assess_stop\nlength_analysis_outer <- lookback + 1\nlength_assess <- assess_stop - assess_start + 1\n\n# relative sizes\nprop_analysis <- length_analysis_outer / length_window\nprop_assess <- length_assess / length_window\n\n# absolute sizes in the inner split\nlength_analysis_inner <- ceiling(prop_analysis * length_analysis_outer)\nlength_calibration <- ceiling(prop_assess * length_analysis_outer)\n\nc(length_analysis_inner, length_calibration)\n#> [1] 3 2\n```\n:::\n\n\nCalculating the length of the calibration set rather than the gap, together with rounding up when translating proportions to new lengths within the outer analysis set means that we prioritize allocating observations to the (inner) analysis and calibration set over allocating them to the gap. In this example here, this means that we are not leaving a gap between the analysis set and the calibration set.\n\nHowever, rounding up for both (inner) analysis and calibration set when we don't have a gap could mean we end up allocating more observations than we actually have. So in that case, we try to take from the calibration set if possible and thus prioritzing fitting the prediction model over the calibration model.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nif (length_analysis_inner + length_calibration > length_analysis_outer) {\n if (length_calibration > 1) {\n length_calibration <- length_calibration - 1\n } else {\n length_analysis_inner <- length_analysis_inner - 1\n }\n}\n```\n:::\n\n\nKeep in mind though that this example dataset is pathalogically small for illustration purposes. Of these details, the prioritization of analysis and calibration set over any gap in between them is most likely to be relevant. But even this should not be massively influential for amounts of data typically required for calibration.\n\nNote that we cannot create a calibration split if `lookback = 0`, as this means that we only have a single row to work with.\n\nFrom the `length_*`s, we can calculate the new `lookback`, `assess_start`, and `assess_stop` to be applied to the outer analysis set.\n\n![](images/calibration-split-window-inner.jpg)\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nsplit_cal <- internal_calibration_split(r_split, .get_split_args(r_set))\nsplit_cal\n#> \n#> <3/2/5>\nanalysis(split_cal)\n#> x\n#> 1 1\n#> 2 2\n#> 3 3\ncalibration(split_cal)\n#> x\n#> 1 4\n#> 2 5\n```\n:::\n\n\nThis is the basic idea for how we split sliding resamples. \n\nApart from the arguments to define the look forwards and backwards, `sliding_window()` has a few more options: `step`, `skip`, and `complete`. The options `step` and `skip` apply on the level of resamples, so we are not applying them to the (inner) split. With `complete = FALSE`, `sliding_window()` allows you to create incomplete analysis sets (but not assessment sets).\n\n![](images/incomplete-window.jpg)\n\nOr, in code:\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\ndf <- data.frame(x = 1:11)\nr_set <- sliding_window(\n df,\n lookback = 4,\n assess_start = 3,\n assess_stop = 5,\n complete = FALSE\n)\nr_set\n#> # Sliding window resampling \n#> # A tibble: 6 × 2\n#> splits id \n#> \n#> 1 Slice1\n#> 2 Slice2\n#> 3 Slice3\n#> 4 Slice4\n#> 5 Slice5\n#> 6 Slice6\n```\n:::\n\n\nWhen creating the calibration split, we apply the `complete` option to the (now inner) analysis set and ensure that the calibration set is complete. In this example, it means that, until we have at least three rows (two for the complete calibration set, and at least one for the analysis set), we can't do a calibration split according to the rules laid out above. This is the case for the second resample, where we only have two rows to work with in the outer analysis set. Instead of erroring here, we fall back to not using calibration at all and return the (outer) analysis set as the (inner) analsysis set, along with an empty calibration set.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 2)\nsplit_cal <- internal_calibration_split(r_split, .get_split_args(r_set))\n#> Warning: Cannot create calibration split; creating an empty calibration set.\n\nsplit_cal\n#> \n#> <2/0/2>\n```\n:::\n\n\nA calibration split on the third resample, which contains three rows, succeeds with the required two rows in the calibration set and the remaining one in the incomplete analysis set.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 3)\nsplit_cal <- internal_calibration_split(r_split, .get_split_args(r_set))\nsplit_cal\n#> \n#> <1/2/3>\n```\n:::\n\n\nIf we can fit a complete (inner) analysis set in the available rows, we do so, even if incomplete sets are allowed.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 5)\nsplit_cal <- internal_calibration_split(r_split, .get_split_args(r_set))\nsplit_cal\n#> \n#> <3/2/5>\n```\n:::\n\n\n\n### Index-based splitting\n\nWhen working with irregular series, it can be helpful to define resamples based on sliding over an index, rather than the rows directly. That index often is a form of time but it does not have to be. Let's take our previous data frame and use `x` as the index, with a small modification. The index doesn't go straight from 1 to 11 but rather is missing an observation at the index value of 2. We use the same `lookback`, `assess_start`, and `assess_stop`. The difference is that now these define the resamples on the _index_ rather than the _rows_ directly.\n\n![](images/calibration-split-index.jpg)\n\nWe still get two resamples, however, the analysis set contains only 4 rows because only those fall into the window defined by the index.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\ndf <- data.frame(x = c(1, 3:11))\nr_set <- sliding_index(\n df,\n index = x,\n lookback = 4,\n assess_start = 3,\n assess_stop = 5\n)\nr_set\n#> # Sliding index resampling \n#> # A tibble: 2 × 2\n#> splits id \n#> \n#> 1 Slice1\n#> 2 Slice2\n```\n:::\n\n\nFor the first resample, all works well. We just skip over the missing row, both when constructing the split into analysis and assessment set and the split into analysis and calibration set.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 1)\nanalysis(r_split)\n#> x\n#> 1 1\n#> 2 3\n#> 3 4\n#> 4 5\n\nsplit_cal <- internal_calibration_split(r_split, .get_split_args(r_set))\nsplit_cal\n#> \n#> <2/2/4>\nanalysis(split_cal)\n#> x\n#> 1 1\n#> 2 3\ncalibration(split_cal)\n#> x\n#> 1 4\n#> 2 5\n```\n:::\n\n\n#### When we can't make a calibration split\n\nAbstracting away from observed rows to an index does pose two challenges for us though.\n\nFor the second resample, the theoretical window for the (outer) analysis set is 2-6. Because it fits within the boundaries of the observed rows (which is 1 to 11), we construct this set, even though we don't observe anything at an index of 2. \n\nBased on `lookback`, `assess_start`, and `assess_stop`, we calculate the same allocation for the (inner) analysis set and the calibration set as in the previous example, when working on the rows directly with `sliding_window()`: three for the inner analysis set and two for the calibration set. However, the dataset we want to split only covers the range [3, 6].\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 2)\nanalysis(r_split)\n#> x\n#> 1 3\n#> 2 4\n#> 3 5\n#> 4 6\n```\n:::\n\n\nThe sliding splits slide over _the data_, meaning they slide over observed values of the index and they slide only within the boundaries of the observed index values. So here, we can only slide within [3, 6] and thus cannot fit an inner analysis set of three and a calibration set of two into it. As established earlier, we fall back onto an empty calibration set in such a situation.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\ninternal_calibration_split(r_split, .get_split_args(r_set))\n#> Warning: Cannot create calibration split; creating an empty calibration set.\n#> \n#> <4/0/4>\n```\n:::\n\n\nSo while we do not observe the lower boundary of our theoretical [2, 6] window from the (outer) analysis set, we always observed the upper boundary. That is because \"sliding over the observed instances\" also means that we will skip over potential splits if we do not observe the value relative to which a split is defined.\n\nIn our example so far, we've been able to generate two resamples on an index running from 1 to 11. The first resample is defined relative to an index of 5, the second one on 6. If there is no observed index of 6, the windows can't be defined and there is no corresponding split and resample.\n\n![](images/calibration-split-index-6.jpg)\n\nOr, in code:\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\n# no index 6\ndf <- data.frame(x = c(1:5, 7:11))\nr_set <- sliding_index(\n df,\n index = x,\n lookback = 4,\n assess_start = 3,\n assess_stop = 5\n)\nr_set\n#> # Sliding index resampling \n#> # A tibble: 1 × 2\n#> splits id \n#> \n#> 1 Slice1\n```\n:::\n\n\nThe one resample we get is relative to an index value of 5, the last observation in the analysis set.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nr_split <- get_rsplit(r_set, 1)\nanalysis(r_split)\n#> x\n#> 1 1\n#> 2 2\n#> 3 3\n#> 4 4\n#> 5 5\n```\n:::\n\n\nThis principle of sliding across _observations_ makes it impossible to construct a calibration split if we happen to define it relative to an index value that we do not observe. Let's modify our example so that now we don't observe anything at index 3.\n\n![](images/calibration-split-index-3.jpg)\n\nThis is the index value relative to which we can define an inner analysis set of 3 and a calibration set of 2. Thus we cannot actually make that split and fall back on an empty calibration set.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\n# missing 3 is the anchor for the inner split of the first resample\ndf <- data.frame(x = c(1:2, 4:11))\nr_set <- sliding_index(\n df,\n index = x,\n lookback = 4,\n assess_start = 3,\n assess_stop = 5\n)\nr_split <- get_rsplit(r_set, 1)\n\ninternal_calibration_split(r_split, .get_split_args(r_set))\n#> Warning: Cannot create calibration split; creating an empty calibration set.\n#> \n#> <4/0/4>\n```\n:::\n\n\nSliding over the observed index values means that we cannot make a calibration split in two situations:\n\n- We don't observe the index value at the lower boundary of the (outer) analysis set.\n- We don't observe the index value relative to which the calibration split would be defined.\n\nIn these situations, we fall back onto an empty calibration set rather than erroring.\n\nWe expect this to not be relevant in practice too often, as chances of this happening are lower with more observations and we assume you are only attempting calibration with sufficiently large datasets.\n\n\n### Period-based splitting\n\nFor the third sliding function in the family, we first aggregate the index to periods and then slide across those, e.g., aggregate daily data to monthly periods. \n\n![](images/calibration-split-period.jpg)\n\nThe principle of how to contruct a calibration split on the (outer) analysis set remains the same. The challenges of abstracting away from the rows, as illustrated for sliding over observed instances of an index also remain. Here, we slide over observed periods. We observe a period, if we observe an index within that period. \n\nFor `lookback = 0`, we remain unable to make a calibration split. However, it is possible to choose a smaller period when defining the resamples, e.g., using weeks instead of months when aggregating daily data. In that case, the calibration split is also defined in terms of weeks, rather than months, and thus failing to split a single month.\n\nNow that we've looked at the principles at play here on very small datasets, let's look at a dataset with a sufficient amount of observations to consider using calibration. The `Chicago` dataset has daily ridership numbers for over 15 years. Here we are constructing resamples which use 2 weeks worth of data for the assessment set, and roughly 15 years (52*15 weeks) for the analysis set. That's enough to use some of those observations for a calibration set!\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\ndata(Chicago, package = \"modeldata\")\n\nchicago_split <- initial_time_split(Chicago, prop = 1 - (14 / nrow(Chicago)))\nchicago_train <- training(chicago_split)\n\nchicago_r_set <- sliding_period(\n chicago_train,\n index = \"date\",\n period = \"week\",\n lookback = 52 * 15,\n assess_stop = 2,\n step = 2\n)\nchicago_r_set\n#> # Sliding period resampling \n#> # A tibble: 16 × 2\n#> splits id \n#> \n#> 1 Slice01\n#> 2 Slice02\n#> 3 Slice03\n#> 4 Slice04\n#> 5 Slice05\n#> 6 Slice06\n#> 7 Slice07\n#> 8 Slice08\n#> 9 Slice09\n#> 10 Slice10\n#> 11 Slice11\n#> 12 Slice12\n#> 13 Slice13\n#> 14 Slice14\n#> 15 Slice15\n#> 16 Slice16\n```\n:::\n\n\nTaking the first resample, we can see the two weeks for the assessment set.\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nchicago_r_split <- get_rsplit(chicago_r_set, 1)\nchicago_assess <- assessment(chicago_r_split)\nrange(chicago_assess$date)\n#> [1] \"2016-01-07\" \"2016-01-20\"\n```\n:::\n\n\nThe calibration set consists of one week from the (outer) analysis set. \n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nsplit_cal <- internal_calibration_split(\n chicago_r_split,\n .get_split_args(chicago_r_set)\n)\nchicago_cal <- calibration(split_cal)\nrange(chicago_cal$date)\n#> [1] \"2015-12-31\" \"2016-01-06\"\n```\n:::\n\n\n## Splits for bootstrap samples\n\nFor bootstrap samples, our goal of avoiding overfitting means we need to refine our approach to making calibration splits a little further. \n\nIf our (outer) analysis set is a bootstrap sample of the training data, it likely contains several replications of an observation. In the illustration below, the first row of the original data is sampled twice into the (outer) analysis set, the second row three times, and the fourth row also twice.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](images/bootstrap-resample.jpg){fig-align='center' width=700}\n:::\n:::\n\n\nWe don't want those duplicates to be split up into the inner analysis and the calibration set. Simply creating another bootstrap resample from the (outer) analysis set makes it possible for this to happen. \n\nIn the example illustration below, the third row of the (outer) analysis set is labeled `2-1`, as it is the first intance of row 2 from the original data. The other two instances of this row in the (outer) analysis set are labeled `2-2` and `2-3`. In the example, the `2-1` row gets sampled into the (inner) analysis set twice, while `2-2` and `2-3` are not sampled and thus end up in the calibration set. This means that row 2 from the original data is now in both the (inner) analysis set and the calibration set.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](images/bootstrap-regular.jpg){fig-align='center' width=700}\n:::\n:::\n\n\nTo avoid this, we could do a grouped resampling with the row ID as the group. Then all replications of a row would end up in the same set, either (inner) analysis or calibration. However, this would mean that rows in the calibration set are potentially not unique, unlike the typical bootstrap OOB sample.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](images/bootstrap-grouped.jpg){fig-align='center' width=700}\n:::\n:::\n\n\nTo prevent both these potential issues, we create the calibration split by sampling the (inner) analysis set, with replacement, from the pool of unique rows in the (outer) analysis set. The calibration set then consists of the rows in the (outer) analysis set that were not sampled into the (inner) analysis set, like for a typicall OOB sample.\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](images/bootstrap-modified.jpg){fig-align='center' width=700}\n:::\n:::\n\n\n## Summary\n\nWhen using calibration, we typically need generous amounts of data because we need separate data for each of these tasks: fit the primary model, fit the calibration model, and assess performance. \nWhen tuning or resampling a workflow with calibration, tidymodels will automatically turn classic resamples (consisting of an analysis and an assessment set) into resamples which include a calibration set in addition to the analysis and assessment set. This is done according to the following principles: \n\n- The assessment set remains untouched, while the analysis set is split into an (inner) analysis set and a calibration set. \n- The inner calibration split mimicks the outer split (into analysis and assesment set) as closely as possible.\n- If we can't make a calibration split based on these basic principles, we skip the calibration.\n\nFor sliding splits of ordered data, applying those principles is a bit more complex than for other types of splits as the outer split into analysis and assessment is already a bit more complex. We've laid out the details of this here for reference.\nFor bootstrap splits, we don't directly split the (outer) analysis set but rather sample the (inner) analysis set from the unique rows in the (outer) analysis set to avoid data leakage between (inner) analysis and calibration set.\n\nIf you need to make manual calibration splits, please leave us a [feature request](https://github.com/tidymodels/rsample/issues/new?template=feature_request.md).\n", + "supporting": [], + "filters": [ + "rmarkdown/pagebreak.lua" + ], + "includes": {}, + "engineDependencies": {}, + "preserve": {}, + "postProcess": true + } +} \ No newline at end of file diff --git a/learn/index-listing.json b/learn/index-listing.json index 6718339b..ebb1a363 100644 --- a/learn/index-listing.json +++ b/learn/index-listing.json @@ -5,6 +5,7 @@ "/learn/statistics/survival-metrics-details/index.html", "/learn/models/calibration/index.html", "/learn/work/fairness-detectors/index.html", + "/learn/work/calibration-splits/index.html", "/learn/statistics/bootstrap/index.html", "/start/models/index.html", "/learn/models/parsnip-nnet/index.html", diff --git a/learn/work/calibration-splits/images/analysis-calibration-assessment.jpg b/learn/work/calibration-splits/images/analysis-calibration-assessment.jpg new file mode 100644 index 00000000..6943038e Binary files /dev/null and b/learn/work/calibration-splits/images/analysis-calibration-assessment.jpg differ diff --git a/learn/work/calibration-splits/images/bootstrap-grouped.jpg b/learn/work/calibration-splits/images/bootstrap-grouped.jpg new file mode 100644 index 00000000..215c4bd4 Binary files /dev/null and b/learn/work/calibration-splits/images/bootstrap-grouped.jpg differ diff --git a/learn/work/calibration-splits/images/bootstrap-modified.jpg b/learn/work/calibration-splits/images/bootstrap-modified.jpg new file mode 100644 index 00000000..2bc2a670 Binary files /dev/null and b/learn/work/calibration-splits/images/bootstrap-modified.jpg differ diff --git a/learn/work/calibration-splits/images/bootstrap-regular.jpg b/learn/work/calibration-splits/images/bootstrap-regular.jpg new file mode 100644 index 00000000..80e083de Binary files /dev/null and b/learn/work/calibration-splits/images/bootstrap-regular.jpg differ diff --git a/learn/work/calibration-splits/images/bootstrap-resample.jpg b/learn/work/calibration-splits/images/bootstrap-resample.jpg new file mode 100644 index 00000000..28bcde53 Binary files /dev/null and b/learn/work/calibration-splits/images/bootstrap-resample.jpg differ diff --git a/learn/work/calibration-splits/images/calibration-split-index-3.jpg b/learn/work/calibration-splits/images/calibration-split-index-3.jpg new file mode 100644 index 00000000..143bc542 Binary files /dev/null and b/learn/work/calibration-splits/images/calibration-split-index-3.jpg differ diff --git a/learn/work/calibration-splits/images/calibration-split-index-6.jpg b/learn/work/calibration-splits/images/calibration-split-index-6.jpg new file mode 100644 index 00000000..b7e38a9c Binary files /dev/null and b/learn/work/calibration-splits/images/calibration-split-index-6.jpg differ diff --git a/learn/work/calibration-splits/images/calibration-split-index.jpg b/learn/work/calibration-splits/images/calibration-split-index.jpg new file mode 100644 index 00000000..4d3b0ea8 Binary files /dev/null and b/learn/work/calibration-splits/images/calibration-split-index.jpg differ diff --git a/learn/work/calibration-splits/images/calibration-split-period.jpg b/learn/work/calibration-splits/images/calibration-split-period.jpg new file mode 100644 index 00000000..bf688cc4 Binary files /dev/null and b/learn/work/calibration-splits/images/calibration-split-period.jpg differ diff --git a/learn/work/calibration-splits/images/calibration-split-window-inner.jpg b/learn/work/calibration-splits/images/calibration-split-window-inner.jpg new file mode 100644 index 00000000..c0cf73ac Binary files /dev/null and b/learn/work/calibration-splits/images/calibration-split-window-inner.jpg differ diff --git a/learn/work/calibration-splits/images/calibration-split-window.jpg b/learn/work/calibration-splits/images/calibration-split-window.jpg new file mode 100644 index 00000000..6ab0f5ec Binary files /dev/null and b/learn/work/calibration-splits/images/calibration-split-window.jpg differ diff --git a/learn/work/calibration-splits/images/incomplete-window.jpg b/learn/work/calibration-splits/images/incomplete-window.jpg new file mode 100644 index 00000000..218d925f Binary files /dev/null and b/learn/work/calibration-splits/images/incomplete-window.jpg differ diff --git a/learn/work/calibration-splits/index.html.md b/learn/work/calibration-splits/index.html.md new file mode 100644 index 00000000..70aed643 --- /dev/null +++ b/learn/work/calibration-splits/index.html.md @@ -0,0 +1,575 @@ +--- +title: "Automatic calibration splits for resamples" +categories: + - post-processing + - calibration + - rsample +type: learn-subsection +weight: 6 +description: | + Learn how tidymodels generates automatic calibration sets for post-processing. +toc: true +toc-depth: 2 +include-after-body: ../../../resources.html +--- + +While preprocessing is the transformation of the predictors prior to a model fit, post-processing is the transformation of the predictions after the model fit. This could be as straightforward as limiting predictions to a certain range of values to as complicated as transforming them based on a separate calibration model. + +A calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to accidentially overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed). + +rsample provides a collection of functions to make resamples for empirical validation of prediction models. So far, the assumption was that the prediction model is the only model that needs fitting, i.e., a resample consists of an analysis set and an assessment set. + +If we include calibration into our workflow (bundeling preprocessing, (primary) model, and post-processing), we want an analysis set, a calibration set, and an assessment set. + +Here we describe how we create these sets automatically when you resample or tune a workflow with calibration. Note that you currently cannot create manual calibration splits and supply them to the functions in tune, like `fit_resamples()` or `tune_grid()`. If you want to do this, leave us a [feature request](https://github.com/tidymodels/rsample/issues/new?template=feature_request.md). + +## General principles + +First principle: we leave the assessment set purely for performance assessment and rather treat the analysis set as "the data to fit" - except now we split it to fit two different models instead of just one. Or in other words, we take the data for the calibration set from the analysis set, not from the assessment set. + +If you compare a model with calibration to one without, and you use the same resamples, you are also using the same assessment sets. + +"Taking data from the analysis set" means splitting up the analysis set to end up with ... an analysis set and a calibration set. Now we have two sets called analysis set, that's confusing. If we need to distinguish them, we'll refer to them as "outer" and "inner" analysis set for "before" and "after" the split for a calibration set. + +::: {.cell layout-align="center"} +::: {.cell-output-display} +![](images/analysis-calibration-assessment.jpg){fig-align='center' height=350} +::: +::: + +It is important to note that including calibration requires more data than fitting a workflow without calibration. For small datasets, you might run into situations where fitting either model (primary model or calibration model) is not particularly stable - or you might not even be able to construct a calibration set properly. + +Second principle: we try to mimick the data splitting process used to create the (outer) analysis set and the assessment set. + +And lastly, the third principle: if we cannot make a calibration split, we fall back onto an empty calibration set rather than erroring. + +Let's look at v-fold cross-validation as an example. + +### Example: v-fold cross-validation + +For v-fold cross-validation, we split the data into `v` folds and create `v` resamples by always using one of the folds as the assessment set while combining the other folds into the analysis set. + +Since we want to have a calibration split similar in spirit, we are splitting the (outer) analysis set into (inner) analysis set and calibration set with the same proportion of observations going into the analysis set: 1 - 1/v. + +This tidymodels-internal split of the (outer) analysis set that happens automatically during resampling is implemented in `internal_calibration_split()`. This function has methods for various types of resamples. They are exported to be used by tune but not intended to be user-facing. We are showing them here for illustration purposes. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +set.seed(11) +r_set <- vfold_cv(warpbreaks, v = 5) +r_split <- get_rsplit(r_set, 1) +r_split +#> +#> <43/11/54> + +# proportion of observations allocated for fitting the model in the outer split +nrow(analysis(r_split)) / nrow(warpbreaks) +#> [1] 0.7962963 + +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +#> +#> <34/9/43> + +# similar proportion of observations allocated for fitting the model in the inner split +nrow(analysis(split_cal)) / nrow(analysis(r_split)) +#> [1] 0.7906977 +``` +::: + +For most types of resamples, these principles are straightforward to apply. However, for resamples of ordered observations (like time series data) and bootstrap samples, we need to consider additional aspects. + +## Splits for ordered observations + +rsample's functions for creating resamples of ordered observations are the `sliding_*()` family: `sliding_window()`, `sliding_index()`, and `sliding_period()`. + +- `sliding_window()` operates on the ordered rows, allowing us to specify how far to look back to create an analysis set, how far to look ahead to create an assessment set and then sliding that across the data to create a set of resamples. +- `sliding_index()` works in much the same way, with the key difference being that what we are sliding over aren't the rows directly but rather an index. This is useful when dealing with irregular series of data: while the number of rows which fall into a window can vary, the window length is the same across all resamples. +- `sliding_period()` takes the idea of the `sliding_index()` function one step further and aggregates the index into periods before sliding across. This is useful for aggregating, e.g., daily data into months and then defining the analysis and assessment set in terms of months rather than the daily index directly. + +### Row-based splitting + +Let's start with the row-based splitting done by `sliding_window()`. We'll use a very small example dataset. This will make it easier to illustrate how the different subsets of the data are created but note that it is too small for real-world purposes. Let's use a data frame with 11 rows and say we want to use 5 for the analysis set, 3 for the assessment set, and leave a gap of 2 in between those two sets. We can make two such resamples from our data frame. + +![](images/calibration-split-window.jpg) + +Or, in code: + +::: {.cell layout-align="center"} + +```{.r .cell-code} +df <- data.frame(x = 1:11) +r_set <- sliding_window(df, lookback = 4, assess_start = 3, assess_stop = 5) +r_set +#> # Sliding window resampling +#> # A tibble: 2 × 2 +#> splits id +#> +#> 1 Slice1 +#> 2 Slice2 +``` +::: + +Let's take that first resample and split it into an (inner) analysis set and a calibration set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 1) +analysis(r_split) +#> x +#> 1 1 +#> 2 2 +#> 3 3 +#> 4 4 +#> 5 5 +``` +::: + +To mimick the split into (outer) analysis and assessment set, we'll calculate the proportion of the analysis set and apply it to the observations available for fitting both the primary model and the calibration model, creating our (inner) analysis set. Since the remaining observations could be for the calibration model or a gap, we also calculate the proportion of the assessment set and use it to construct the calibration set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +lookback <- 4 +assess_start <- 3 +assess_stop <- 5 + +# absolute sizes in the outer split +length_window <- lookback + 1 + assess_stop +length_analysis_outer <- lookback + 1 +length_assess <- assess_stop - assess_start + 1 + +# relative sizes +prop_analysis <- length_analysis_outer / length_window +prop_assess <- length_assess / length_window + +# absolute sizes in the inner split +length_analysis_inner <- ceiling(prop_analysis * length_analysis_outer) +length_calibration <- ceiling(prop_assess * length_analysis_outer) + +c(length_analysis_inner, length_calibration) +#> [1] 3 2 +``` +::: + +Calculating the length of the calibration set rather than the gap, together with rounding up when translating proportions to new lengths within the outer analysis set means that we prioritize allocating observations to the (inner) analysis and calibration set over allocating them to the gap. In this example here, this means that we are not leaving a gap between the analysis set and the calibration set. + +However, rounding up for both (inner) analysis and calibration set when we don't have a gap could mean we end up allocating more observations than we actually have. So in that case, we try to take from the calibration set if possible and thus prioritzing fitting the prediction model over the calibration model. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +if (length_analysis_inner + length_calibration > length_analysis_outer) { + if (length_calibration > 1) { + length_calibration <- length_calibration - 1 + } else { + length_analysis_inner <- length_analysis_inner - 1 + } +} +``` +::: + +Keep in mind though that this example dataset is pathalogically small for illustration purposes. Of these details, the prioritization of analysis and calibration set over any gap in between them is most likely to be relevant. But even this should not be massively influential for amounts of data typically required for calibration. + +Note that we cannot create a calibration split if `lookback = 0`, as this means that we only have a single row to work with. + +From the `length_*`s, we can calculate the new `lookback`, `assess_start`, and `assess_stop` to be applied to the outer analysis set. + +![](images/calibration-split-window-inner.jpg) + +::: {.cell layout-align="center"} + +```{.r .cell-code} +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +#> +#> <3/2/5> +analysis(split_cal) +#> x +#> 1 1 +#> 2 2 +#> 3 3 +calibration(split_cal) +#> x +#> 1 4 +#> 2 5 +``` +::: + +This is the basic idea for how we split sliding resamples. + +Apart from the arguments to define the look forwards and backwards, `sliding_window()` has a few more options: `step`, `skip`, and `complete`. The options `step` and `skip` apply on the level of resamples, so we are not applying them to the (inner) split. With `complete = FALSE`, `sliding_window()` allows you to create incomplete analysis sets (but not assessment sets). + +![](images/incomplete-window.jpg) + +Or, in code: + +::: {.cell layout-align="center"} + +```{.r .cell-code} +df <- data.frame(x = 1:11) +r_set <- sliding_window( + df, + lookback = 4, + assess_start = 3, + assess_stop = 5, + complete = FALSE +) +r_set +#> # Sliding window resampling +#> # A tibble: 6 × 2 +#> splits id +#> +#> 1 Slice1 +#> 2 Slice2 +#> 3 Slice3 +#> 4 Slice4 +#> 5 Slice5 +#> 6 Slice6 +``` +::: + +When creating the calibration split, we apply the `complete` option to the (now inner) analysis set and ensure that the calibration set is complete. In this example, it means that, until we have at least three rows (two for the complete calibration set, and at least one for the analysis set), we can't do a calibration split according to the rules laid out above. This is the case for the second resample, where we only have two rows to work with in the outer analysis set. Instead of erroring here, we fall back to not using calibration at all and return the (outer) analysis set as the (inner) analsysis set, along with an empty calibration set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 2) +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +#> Warning: Cannot create calibration split; creating an empty calibration set. + +split_cal +#> +#> <2/0/2> +``` +::: + +A calibration split on the third resample, which contains three rows, succeeds with the required two rows in the calibration set and the remaining one in the incomplete analysis set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 3) +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +#> +#> <1/2/3> +``` +::: + +If we can fit a complete (inner) analysis set in the available rows, we do so, even if incomplete sets are allowed. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 5) +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +#> +#> <3/2/5> +``` +::: + +### Index-based splitting + +When working with irregular series, it can be helpful to define resamples based on sliding over an index, rather than the rows directly. That index often is a form of time but it does not have to be. Let's take our previous data frame and use `x` as the index, with a small modification. The index doesn't go straight from 1 to 11 but rather is missing an observation at the index value of 2. We use the same `lookback`, `assess_start`, and `assess_stop`. The difference is that now these define the resamples on the _index_ rather than the _rows_ directly. + +![](images/calibration-split-index.jpg) + +We still get two resamples, however, the analysis set contains only 4 rows because only those fall into the window defined by the index. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +df <- data.frame(x = c(1, 3:11)) +r_set <- sliding_index( + df, + index = x, + lookback = 4, + assess_start = 3, + assess_stop = 5 +) +r_set +#> # Sliding index resampling +#> # A tibble: 2 × 2 +#> splits id +#> +#> 1 Slice1 +#> 2 Slice2 +``` +::: + +For the first resample, all works well. We just skip over the missing row, both when constructing the split into analysis and assessment set and the split into analysis and calibration set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 1) +analysis(r_split) +#> x +#> 1 1 +#> 2 3 +#> 3 4 +#> 4 5 + +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +#> +#> <2/2/4> +analysis(split_cal) +#> x +#> 1 1 +#> 2 3 +calibration(split_cal) +#> x +#> 1 4 +#> 2 5 +``` +::: + +#### When we can't make a calibration split + +Abstracting away from observed rows to an index does pose two challenges for us though. + +For the second resample, the theoretical window for the (outer) analysis set is 2-6. Because it fits within the boundaries of the observed rows (which is 1 to 11), we construct this set, even though we don't observe anything at an index of 2. + +Based on `lookback`, `assess_start`, and `assess_stop`, we calculate the same allocation for the (inner) analysis set and the calibration set as in the previous example, when working on the rows directly with `sliding_window()`: three for the inner analysis set and two for the calibration set. However, the dataset we want to split only covers the range [3, 6]. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 2) +analysis(r_split) +#> x +#> 1 3 +#> 2 4 +#> 3 5 +#> 4 6 +``` +::: + +The sliding splits slide over _the data_, meaning they slide over observed values of the index and they slide only within the boundaries of the observed index values. So here, we can only slide within [3, 6] and thus cannot fit an inner analysis set of three and a calibration set of two into it. As established earlier, we fall back onto an empty calibration set in such a situation. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +internal_calibration_split(r_split, .get_split_args(r_set)) +#> Warning: Cannot create calibration split; creating an empty calibration set. +#> +#> <4/0/4> +``` +::: + +So while we do not observe the lower boundary of our theoretical [2, 6] window from the (outer) analysis set, we always observed the upper boundary. That is because "sliding over the observed instances" also means that we will skip over potential splits if we do not observe the value relative to which a split is defined. + +In our example so far, we've been able to generate two resamples on an index running from 1 to 11. The first resample is defined relative to an index of 5, the second one on 6. If there is no observed index of 6, the windows can't be defined and there is no corresponding split and resample. + +![](images/calibration-split-index-6.jpg) + +Or, in code: + +::: {.cell layout-align="center"} + +```{.r .cell-code} +# no index 6 +df <- data.frame(x = c(1:5, 7:11)) +r_set <- sliding_index( + df, + index = x, + lookback = 4, + assess_start = 3, + assess_stop = 5 +) +r_set +#> # Sliding index resampling +#> # A tibble: 1 × 2 +#> splits id +#> +#> 1 Slice1 +``` +::: + +The one resample we get is relative to an index value of 5, the last observation in the analysis set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +r_split <- get_rsplit(r_set, 1) +analysis(r_split) +#> x +#> 1 1 +#> 2 2 +#> 3 3 +#> 4 4 +#> 5 5 +``` +::: + +This principle of sliding across _observations_ makes it impossible to construct a calibration split if we happen to define it relative to an index value that we do not observe. Let's modify our example so that now we don't observe anything at index 3. + +![](images/calibration-split-index-3.jpg) + +This is the index value relative to which we can define an inner analysis set of 3 and a calibration set of 2. Thus we cannot actually make that split and fall back on an empty calibration set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +# missing 3 is the anchor for the inner split of the first resample +df <- data.frame(x = c(1:2, 4:11)) +r_set <- sliding_index( + df, + index = x, + lookback = 4, + assess_start = 3, + assess_stop = 5 +) +r_split <- get_rsplit(r_set, 1) + +internal_calibration_split(r_split, .get_split_args(r_set)) +#> Warning: Cannot create calibration split; creating an empty calibration set. +#> +#> <4/0/4> +``` +::: + +Sliding over the observed index values means that we cannot make a calibration split in two situations: + +- We don't observe the index value at the lower boundary of the (outer) analysis set. +- We don't observe the index value relative to which the calibration split would be defined. + +In these situations, we fall back onto an empty calibration set rather than erroring. + +We expect this to not be relevant in practice too often, as chances of this happening are lower with more observations and we assume you are only attempting calibration with sufficiently large datasets. + +### Period-based splitting + +For the third sliding function in the family, we first aggregate the index to periods and then slide across those, e.g., aggregate daily data to monthly periods. + +![](images/calibration-split-period.jpg) + +The principle of how to contruct a calibration split on the (outer) analysis set remains the same. The challenges of abstracting away from the rows, as illustrated for sliding over observed instances of an index also remain. Here, we slide over observed periods. We observe a period, if we observe an index within that period. + +For `lookback = 0`, we remain unable to make a calibration split. However, it is possible to choose a smaller period when defining the resamples, e.g., using weeks instead of months when aggregating daily data. In that case, the calibration split is also defined in terms of weeks, rather than months, and thus failing to split a single month. + +Now that we've looked at the principles at play here on very small datasets, let's look at a dataset with a sufficient amount of observations to consider using calibration. The `Chicago` dataset has daily ridership numbers for over 15 years. Here we are constructing resamples which use 2 weeks worth of data for the assessment set, and roughly 15 years (52*15 weeks) for the analysis set. That's enough to use some of those observations for a calibration set! + +::: {.cell layout-align="center"} + +```{.r .cell-code} +data(Chicago, package = "modeldata") + +chicago_split <- initial_time_split(Chicago, prop = 1 - (14 / nrow(Chicago))) +chicago_train <- training(chicago_split) + +chicago_r_set <- sliding_period( + chicago_train, + index = "date", + period = "week", + lookback = 52 * 15, + assess_stop = 2, + step = 2 +) +chicago_r_set +#> # Sliding period resampling +#> # A tibble: 16 × 2 +#> splits id +#> +#> 1 Slice01 +#> 2 Slice02 +#> 3 Slice03 +#> 4 Slice04 +#> 5 Slice05 +#> 6 Slice06 +#> 7 Slice07 +#> 8 Slice08 +#> 9 Slice09 +#> 10 Slice10 +#> 11 Slice11 +#> 12 Slice12 +#> 13 Slice13 +#> 14 Slice14 +#> 15 Slice15 +#> 16 Slice16 +``` +::: + +Taking the first resample, we can see the two weeks for the assessment set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +chicago_r_split <- get_rsplit(chicago_r_set, 1) +chicago_assess <- assessment(chicago_r_split) +range(chicago_assess$date) +#> [1] "2016-01-07" "2016-01-20" +``` +::: + +The calibration set consists of one week from the (outer) analysis set. + +::: {.cell layout-align="center"} + +```{.r .cell-code} +split_cal <- internal_calibration_split( + chicago_r_split, + .get_split_args(chicago_r_set) +) +chicago_cal <- calibration(split_cal) +range(chicago_cal$date) +#> [1] "2015-12-31" "2016-01-06" +``` +::: + +## Splits for bootstrap samples + +For bootstrap samples, our goal of avoiding overfitting means we need to refine our approach to making calibration splits a little further. + +If our (outer) analysis set is a bootstrap sample of the training data, it likely contains several replications of an observation. In the illustration below, the first row of the original data is sampled twice into the (outer) analysis set, the second row three times, and the fourth row also twice. + +::: {.cell layout-align="center"} +::: {.cell-output-display} +![](images/bootstrap-resample.jpg){fig-align='center' width=700} +::: +::: + +We don't want those duplicates to be split up into the inner analysis and the calibration set. Simply creating another bootstrap resample from the (outer) analysis set makes it possible for this to happen. + +In the example illustration below, the third row of the (outer) analysis set is labeled `2-1`, as it is the first intance of row 2 from the original data. The other two instances of this row in the (outer) analysis set are labeled `2-2` and `2-3`. In the example, the `2-1` row gets sampled into the (inner) analysis set twice, while `2-2` and `2-3` are not sampled and thus end up in the calibration set. This means that row 2 from the original data is now in both the (inner) analysis set and the calibration set. + +::: {.cell layout-align="center"} +::: {.cell-output-display} +![](images/bootstrap-regular.jpg){fig-align='center' width=700} +::: +::: + +To avoid this, we could do a grouped resampling with the row ID as the group. Then all replications of a row would end up in the same set, either (inner) analysis or calibration. However, this would mean that rows in the calibration set are potentially not unique, unlike the typical bootstrap OOB sample. + +::: {.cell layout-align="center"} +::: {.cell-output-display} +![](images/bootstrap-grouped.jpg){fig-align='center' width=700} +::: +::: + +To prevent both these potential issues, we create the calibration split by sampling the (inner) analysis set, with replacement, from the pool of unique rows in the (outer) analysis set. The calibration set then consists of the rows in the (outer) analysis set that were not sampled into the (inner) analysis set, like for a typicall OOB sample. + +::: {.cell layout-align="center"} +::: {.cell-output-display} +![](images/bootstrap-modified.jpg){fig-align='center' width=700} +::: +::: + +## Summary + +When using calibration, we typically need generous amounts of data because we need separate data for each of these tasks: fit the primary model, fit the calibration model, and assess performance. +When tuning or resampling a workflow with calibration, tidymodels will automatically turn classic resamples (consisting of an analysis and an assessment set) into resamples which include a calibration set in addition to the analysis and assessment set. This is done according to the following principles: + +- The assessment set remains untouched, while the analysis set is split into an (inner) analysis set and a calibration set. +- The inner calibration split mimicks the outer split (into analysis and assesment set) as closely as possible. +- If we can't make a calibration split based on these basic principles, we skip the calibration. + +For sliding splits of ordered data, applying those principles is a bit more complex than for other types of splits as the outer split into analysis and assessment is already a bit more complex. We've laid out the details of this here for reference. +For bootstrap splits, we don't directly split the (outer) analysis set but rather sample the (inner) analysis set from the unique rows in the (outer) analysis set to avoid data leakage between (inner) analysis and calibration set. + +If you need to make manual calibration splits, please leave us a [feature request](https://github.com/tidymodels/rsample/issues/new?template=feature_request.md). diff --git a/learn/work/calibration-splits/index.qmd b/learn/work/calibration-splits/index.qmd new file mode 100644 index 00000000..fefc99be --- /dev/null +++ b/learn/work/calibration-splits/index.qmd @@ -0,0 +1,478 @@ +--- +title: "Automatic calibration splits for resamples" +categories: + - post-processing + - calibration + - rsample +type: learn-subsection +weight: 6 +description: | + Learn how tidymodels generates automatic calibration sets for post-processing. +toc: true +toc-depth: 2 +include-after-body: ../../../resources.html +--- + +```{r} +#| label: "setup" +#| include: false +#| message: false +#| warning: false +source(here::here("common.R")) +``` + +```{r} +#| label: "load" +#| include: false +#| message: false +#| warning: false +library(tidymodels) + +pkgs <- c("tidymodels", "rsample") + +theme_set(theme_bw() + theme(legend.position = "top")) +``` + +While preprocessing is the transformation of the predictors prior to a model fit, post-processing is the transformation of the predictions after the model fit. This could be as straightforward as limiting predictions to a certain range of values to as complicated as transforming them based on a separate calibration model. + +A calibration model is used to model the relationship between the predictions based on the primary model and the true outcomes. An additional model means an additional chance to accidentially overfit. So when working with calibration, this is crucial: we cannot use the same data to fit our calibration model as we use to assess the combination of primary and calibration model. Using the same data to fit the primary model and the calibration model means the predictions used to fit the calibration model are re-predictions of the same observations used to fit the primary model. Hence they are closer to the true values than predictions on new data would be and the calibration model doesn't have accurate information to estimate the right trends (so that they then can be removed). + +rsample provides a collection of functions to make resamples for empirical validation of prediction models. So far, the assumption was that the prediction model is the only model that needs fitting, i.e., a resample consists of an analysis set and an assessment set. + +If we include calibration into our workflow (bundeling preprocessing, (primary) model, and post-processing), we want an analysis set, a calibration set, and an assessment set. + +Here we describe how we create these sets automatically when you resample or tune a workflow with calibration. Note that you currently cannot create manual calibration splits and supply them to the functions in tune, like `fit_resamples()` or `tune_grid()`. If you want to do this, leave us a [feature request](https://github.com/tidymodels/rsample/issues/new?template=feature_request.md). + +## General principles + +First principle: we leave the assessment set purely for performance assessment and rather treat the analysis set as "the data to fit" - except now we split it to fit two different models instead of just one. Or in other words, we take the data for the calibration set from the analysis set, not from the assessment set. + +If you compare a model with calibration to one without, and you use the same resamples, you are also using the same assessment sets. + +"Taking data from the analysis set" means splitting up the analysis set to end up with ... an analysis set and a calibration set. Now we have two sets called analysis set, that's confusing. If we need to distinguish them, we'll refer to them as "outer" and "inner" analysis set for "before" and "after" the split for a calibration set. + + +```{r} +#| label: calibration +#| echo: false +#| out.height: 350 +#| fig.align: "center" +knitr::include_graphics("images/analysis-calibration-assessment.jpg") +``` + +It is important to note that including calibration requires more data than fitting a workflow without calibration. For small datasets, you might run into situations where fitting either model (primary model or calibration model) is not particularly stable - or you might not even be able to construct a calibration set properly. + +Second principle: we try to mimick the data splitting process used to create the (outer) analysis set and the assessment set. + +And lastly, the third principle: if we cannot make a calibration split, we fall back onto an empty calibration set rather than erroring. + +Let's look at v-fold cross-validation as an example. + + +### Example: v-fold cross-validation + +For v-fold cross-validation, we split the data into `v` folds and create `v` resamples by always using one of the folds as the assessment set while combining the other folds into the analysis set. + +Since we want to have a calibration split similar in spirit, we are splitting the (outer) analysis set into (inner) analysis set and calibration set with the same proportion of observations going into the analysis set: 1 - 1/v. + +This tidymodels-internal split of the (outer) analysis set that happens automatically during resampling is implemented in `internal_calibration_split()`. This function has methods for various types of resamples. They are exported to be used by tune but not intended to be user-facing. We are showing them here for illustration purposes. + +```{r} +#| label: vfold + +set.seed(11) +r_set <- vfold_cv(warpbreaks, v = 5) +r_split <- get_rsplit(r_set, 1) +r_split + +# proportion of observations allocated for fitting the model in the outer split +nrow(analysis(r_split)) / nrow(warpbreaks) + +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal + +# similar proportion of observations allocated for fitting the model in the inner split +nrow(analysis(split_cal)) / nrow(analysis(r_split)) +``` + +For most types of resamples, these principles are straightforward to apply. However, for resamples of ordered observations (like time series data) and bootstrap samples, we need to consider additional aspects. + +## Splits for ordered observations + +rsample's functions for creating resamples of ordered observations are the `sliding_*()` family: `sliding_window()`, `sliding_index()`, and `sliding_period()`. + +- `sliding_window()` operates on the ordered rows, allowing us to specify how far to look back to create an analysis set, how far to look ahead to create an assessment set and then sliding that across the data to create a set of resamples. +- `sliding_index()` works in much the same way, with the key difference being that what we are sliding over aren't the rows directly but rather an index. This is useful when dealing with irregular series of data: while the number of rows which fall into a window can vary, the window length is the same across all resamples. +- `sliding_period()` takes the idea of the `sliding_index()` function one step further and aggregates the index into periods before sliding across. This is useful for aggregating, e.g., daily data into months and then defining the analysis and assessment set in terms of months rather than the daily index directly. + +### Row-based splitting + +Let's start with the row-based splitting done by `sliding_window()`. We'll use a very small example dataset. This will make it easier to illustrate how the different subsets of the data are created but note that it is too small for real-world purposes. Let's use a data frame with 11 rows and say we want to use 5 for the analysis set, 3 for the assessment set, and leave a gap of 2 in between those two sets. We can make two such resamples from our data frame. + +![](images/calibration-split-window.jpg) + +Or, in code: +```{r} +#| label: sliding_window_rset + +df <- data.frame(x = 1:11) +r_set <- sliding_window(df, lookback = 4, assess_start = 3, assess_stop = 5) +r_set +``` + +Let's take that first resample and split it into an (inner) analysis set and a calibration set. + +```{r} +#| label: sliding_window_rsplit + +r_split <- get_rsplit(r_set, 1) +analysis(r_split) +``` + +To mimick the split into (outer) analysis and assessment set, we'll calculate the proportion of the analysis set and apply it to the observations available for fitting both the primary model and the calibration model, creating our (inner) analysis set. Since the remaining observations could be for the calibration model or a gap, we also calculate the proportion of the assessment set and use it to construct the calibration set. + +```{r} +#| label: sliding_window_props + +lookback <- 4 +assess_start <- 3 +assess_stop <- 5 + +# absolute sizes in the outer split +length_window <- lookback + 1 + assess_stop +length_analysis_outer <- lookback + 1 +length_assess <- assess_stop - assess_start + 1 + +# relative sizes +prop_analysis <- length_analysis_outer / length_window +prop_assess <- length_assess / length_window + +# absolute sizes in the inner split +length_analysis_inner <- ceiling(prop_analysis * length_analysis_outer) +length_calibration <- ceiling(prop_assess * length_analysis_outer) + +c(length_analysis_inner, length_calibration) +``` + +Calculating the length of the calibration set rather than the gap, together with rounding up when translating proportions to new lengths within the outer analysis set means that we prioritize allocating observations to the (inner) analysis and calibration set over allocating them to the gap. In this example here, this means that we are not leaving a gap between the analysis set and the calibration set. + +However, rounding up for both (inner) analysis and calibration set when we don't have a gap could mean we end up allocating more observations than we actually have. So in that case, we try to take from the calibration set if possible and thus prioritzing fitting the prediction model over the calibration model. + +```{r} +#| label: sliding_window_adjustments + +if (length_analysis_inner + length_calibration > length_analysis_outer) { + if (length_calibration > 1) { + length_calibration <- length_calibration - 1 + } else { + length_analysis_inner <- length_analysis_inner - 1 + } +} +``` + +Keep in mind though that this example dataset is pathalogically small for illustration purposes. Of these details, the prioritization of analysis and calibration set over any gap in between them is most likely to be relevant. But even this should not be massively influential for amounts of data typically required for calibration. + +Note that we cannot create a calibration split if `lookback = 0`, as this means that we only have a single row to work with. + +From the `length_*`s, we can calculate the new `lookback`, `assess_start`, and `assess_stop` to be applied to the outer analysis set. + +![](images/calibration-split-window-inner.jpg) + +```{r} +#| label: sliding_window_internal_calibration_split + +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +analysis(split_cal) +calibration(split_cal) +``` + +This is the basic idea for how we split sliding resamples. + +Apart from the arguments to define the look forwards and backwards, `sliding_window()` has a few more options: `step`, `skip`, and `complete`. The options `step` and `skip` apply on the level of resamples, so we are not applying them to the (inner) split. With `complete = FALSE`, `sliding_window()` allows you to create incomplete analysis sets (but not assessment sets). + +![](images/incomplete-window.jpg) + +Or, in code: + +```{r} +#| label: sliding_window_incomplete + +df <- data.frame(x = 1:11) +r_set <- sliding_window( + df, + lookback = 4, + assess_start = 3, + assess_stop = 5, + complete = FALSE +) +r_set +``` + +When creating the calibration split, we apply the `complete` option to the (now inner) analysis set and ensure that the calibration set is complete. In this example, it means that, until we have at least three rows (two for the complete calibration set, and at least one for the analysis set), we can't do a calibration split according to the rules laid out above. This is the case for the second resample, where we only have two rows to work with in the outer analysis set. Instead of erroring here, we fall back to not using calibration at all and return the (outer) analysis set as the (inner) analsysis set, along with an empty calibration set. + +```{r} +#| label: sliding_window_split_incomplete_2 + +r_split <- get_rsplit(r_set, 2) +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) + +split_cal +``` + +A calibration split on the third resample, which contains three rows, succeeds with the required two rows in the calibration set and the remaining one in the incomplete analysis set. + +```{r} +#| label: sliding_window_split_incomplete_3 + +r_split <- get_rsplit(r_set, 3) +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +``` + +If we can fit a complete (inner) analysis set in the available rows, we do so, even if incomplete sets are allowed. + +```{r} +#| label: sliding_window_split_incomplete_5 + +r_split <- get_rsplit(r_set, 5) +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +``` + + +### Index-based splitting + +When working with irregular series, it can be helpful to define resamples based on sliding over an index, rather than the rows directly. That index often is a form of time but it does not have to be. Let's take our previous data frame and use `x` as the index, with a small modification. The index doesn't go straight from 1 to 11 but rather is missing an observation at the index value of 2. We use the same `lookback`, `assess_start`, and `assess_stop`. The difference is that now these define the resamples on the _index_ rather than the _rows_ directly. + +![](images/calibration-split-index.jpg) + +We still get two resamples, however, the analysis set contains only 4 rows because only those fall into the window defined by the index. + +```{r} +#| label: sliding_index_rset + +df <- data.frame(x = c(1, 3:11)) +r_set <- sliding_index( + df, + index = x, + lookback = 4, + assess_start = 3, + assess_stop = 5 +) +r_set +``` + +For the first resample, all works well. We just skip over the missing row, both when constructing the split into analysis and assessment set and the split into analysis and calibration set. + +```{r} +#| label: sliding_index_rsplit_1 + +r_split <- get_rsplit(r_set, 1) +analysis(r_split) + +split_cal <- internal_calibration_split(r_split, .get_split_args(r_set)) +split_cal +analysis(split_cal) +calibration(split_cal) +``` + +#### When we can't make a calibration split + +Abstracting away from observed rows to an index does pose two challenges for us though. + +For the second resample, the theoretical window for the (outer) analysis set is 2-6. Because it fits within the boundaries of the observed rows (which is 1 to 11), we construct this set, even though we don't observe anything at an index of 2. + +Based on `lookback`, `assess_start`, and `assess_stop`, we calculate the same allocation for the (inner) analysis set and the calibration set as in the previous example, when working on the rows directly with `sliding_window()`: three for the inner analysis set and two for the calibration set. However, the dataset we want to split only covers the range [3, 6]. + +```{r} +#| label: sliding_index_rsplit_2 + +r_split <- get_rsplit(r_set, 2) +analysis(r_split) +``` + +The sliding splits slide over _the data_, meaning they slide over observed values of the index and they slide only within the boundaries of the observed index values. So here, we can only slide within [3, 6] and thus cannot fit an inner analysis set of three and a calibration set of two into it. As established earlier, we fall back onto an empty calibration set in such a situation. + +```{r} +#| label: sliding_index_rsplit_2_fallback + +internal_calibration_split(r_split, .get_split_args(r_set)) +``` + +So while we do not observe the lower boundary of our theoretical [2, 6] window from the (outer) analysis set, we always observed the upper boundary. That is because "sliding over the observed instances" also means that we will skip over potential splits if we do not observe the value relative to which a split is defined. + +In our example so far, we've been able to generate two resamples on an index running from 1 to 11. The first resample is defined relative to an index of 5, the second one on 6. If there is no observed index of 6, the windows can't be defined and there is no corresponding split and resample. + +![](images/calibration-split-index-6.jpg) + +Or, in code: + +```{r} +#| label: sliding_index_rset_anchor + +# no index 6 +df <- data.frame(x = c(1:5, 7:11)) +r_set <- sliding_index( + df, + index = x, + lookback = 4, + assess_start = 3, + assess_stop = 5 +) +r_set +``` + +The one resample we get is relative to an index value of 5, the last observation in the analysis set. + +```{r} +#| label: sliding_index_rsplit_anchor + +r_split <- get_rsplit(r_set, 1) +analysis(r_split) +``` + +This principle of sliding across _observations_ makes it impossible to construct a calibration split if we happen to define it relative to an index value that we do not observe. Let's modify our example so that now we don't observe anything at index 3. + +![](images/calibration-split-index-3.jpg) + +This is the index value relative to which we can define an inner analysis set of 3 and a calibration set of 2. Thus we cannot actually make that split and fall back on an empty calibration set. + +```{r} +#| label: sliding_index_split_anchor + +# missing 3 is the anchor for the inner split of the first resample +df <- data.frame(x = c(1:2, 4:11)) +r_set <- sliding_index( + df, + index = x, + lookback = 4, + assess_start = 3, + assess_stop = 5 +) +r_split <- get_rsplit(r_set, 1) + +internal_calibration_split(r_split, .get_split_args(r_set)) +``` + +Sliding over the observed index values means that we cannot make a calibration split in two situations: + +- We don't observe the index value at the lower boundary of the (outer) analysis set. +- We don't observe the index value relative to which the calibration split would be defined. + +In these situations, we fall back onto an empty calibration set rather than erroring. + +We expect this to not be relevant in practice too often, as chances of this happening are lower with more observations and we assume you are only attempting calibration with sufficiently large datasets. + + +### Period-based splitting + +For the third sliding function in the family, we first aggregate the index to periods and then slide across those, e.g., aggregate daily data to monthly periods. + +![](images/calibration-split-period.jpg) + +The principle of how to contruct a calibration split on the (outer) analysis set remains the same. The challenges of abstracting away from the rows, as illustrated for sliding over observed instances of an index also remain. Here, we slide over observed periods. We observe a period, if we observe an index within that period. + +For `lookback = 0`, we remain unable to make a calibration split. However, it is possible to choose a smaller period when defining the resamples, e.g., using weeks instead of months when aggregating daily data. In that case, the calibration split is also defined in terms of weeks, rather than months, and thus failing to split a single month. + +Now that we've looked at the principles at play here on very small datasets, let's look at a dataset with a sufficient amount of observations to consider using calibration. The `Chicago` dataset has daily ridership numbers for over 15 years. Here we are constructing resamples which use 2 weeks worth of data for the assessment set, and roughly 15 years (52*15 weeks) for the analysis set. That's enough to use some of those observations for a calibration set! + +```{r} +#| label: chicago + +data(Chicago, package = "modeldata") + +chicago_split <- initial_time_split(Chicago, prop = 1 - (14 / nrow(Chicago))) +chicago_train <- training(chicago_split) + +chicago_r_set <- sliding_period( + chicago_train, + index = "date", + period = "week", + lookback = 52 * 15, + assess_stop = 2, + step = 2 +) +chicago_r_set +``` + +Taking the first resample, we can see the two weeks for the assessment set. +```{r} +#| label: chicago_rsplit + +chicago_r_split <- get_rsplit(chicago_r_set, 1) +chicago_assess <- assessment(chicago_r_split) +range(chicago_assess$date) +``` + +The calibration set consists of one week from the (outer) analysis set. +```{r} +#| label: chicago_isplit + +split_cal <- internal_calibration_split( + chicago_r_split, + .get_split_args(chicago_r_set) +) +chicago_cal <- calibration(split_cal) +range(chicago_cal$date) +``` + +## Splits for bootstrap samples + +For bootstrap samples, our goal of avoiding overfitting means we need to refine our approach to making calibration splits a little further. + +If our (outer) analysis set is a bootstrap sample of the training data, it likely contains several replications of an observation. In the illustration below, the first row of the original data is sampled twice into the (outer) analysis set, the second row three times, and the fourth row also twice. + +```{r} +#| label: bootstrap-resample +#| echo: false +#| out.width: 700 +#| fig.align: "center" +knitr::include_graphics("images/bootstrap-resample.jpg") +``` + +We don't want those duplicates to be split up into the inner analysis and the calibration set. Simply creating another bootstrap resample from the (outer) analysis set makes it possible for this to happen. + +In the example illustration below, the third row of the (outer) analysis set is labeled `2-1`, as it is the first intance of row 2 from the original data. The other two instances of this row in the (outer) analysis set are labeled `2-2` and `2-3`. In the example, the `2-1` row gets sampled into the (inner) analysis set twice, while `2-2` and `2-3` are not sampled and thus end up in the calibration set. This means that row 2 from the original data is now in both the (inner) analysis set and the calibration set. + +```{r} +#| label: bootstrap-regular +#| echo: false +#| out.width: 700 +#| fig.align: "center" +knitr::include_graphics("images/bootstrap-regular.jpg") +``` + +To avoid this, we could do a grouped resampling with the row ID as the group. Then all replications of a row would end up in the same set, either (inner) analysis or calibration. However, this would mean that rows in the calibration set are potentially not unique, unlike the typical bootstrap OOB sample. + +```{r} +#| label: bootstrap-grouped +#| echo: false +#| out.width: 700 +#| fig.align: "center" +knitr::include_graphics("images/bootstrap-grouped.jpg") +``` + +To prevent both these potential issues, we create the calibration split by sampling the (inner) analysis set, with replacement, from the pool of unique rows in the (outer) analysis set. The calibration set then consists of the rows in the (outer) analysis set that were not sampled into the (inner) analysis set, like for a typicall OOB sample. + +```{r} +#| label: bootstrap-modified +#| echo: false +#| out.width: 700 +#| fig.align: "center" +knitr::include_graphics("images/bootstrap-modified.jpg") +``` + +## Summary + +When using calibration, we typically need generous amounts of data because we need separate data for each of these tasks: fit the primary model, fit the calibration model, and assess performance. +When tuning or resampling a workflow with calibration, tidymodels will automatically turn classic resamples (consisting of an analysis and an assessment set) into resamples which include a calibration set in addition to the analysis and assessment set. This is done according to the following principles: + +- The assessment set remains untouched, while the analysis set is split into an (inner) analysis set and a calibration set. +- The inner calibration split mimicks the outer split (into analysis and assesment set) as closely as possible. +- If we can't make a calibration split based on these basic principles, we skip the calibration. + +For sliding splits of ordered data, applying those principles is a bit more complex than for other types of splits as the outer split into analysis and assessment is already a bit more complex. We've laid out the details of this here for reference. +For bootstrap splits, we don't directly split the (outer) analysis set but rather sample the (inner) analysis set from the unique rows in the (outer) analysis set to avoid data leakage between (inner) analysis and calibration set. + +If you need to make manual calibration splits, please leave us a [feature request](https://github.com/tidymodels/rsample/issues/new?template=feature_request.md). diff --git a/site_libs/quarto-html/quarto-syntax-highlighting-303f0b4dceb2a814a9bbef461efe1684.css b/site_libs/quarto-html/quarto-syntax-highlighting-303f0b4dceb2a814a9bbef461efe1684.css deleted file mode 100644 index ee5a58e3..00000000 --- a/site_libs/quarto-html/quarto-syntax-highlighting-303f0b4dceb2a814a9bbef461efe1684.css +++ /dev/null @@ -1,236 +0,0 @@ -/* quarto syntax highlight colors */ -:root { - --quarto-hl-ot-color: #003B4F; - --quarto-hl-at-color: #657422; - --quarto-hl-ss-color: #20794D; - --quarto-hl-an-color: #5E5E5E; - --quarto-hl-fu-color: #4758AB; - --quarto-hl-st-color: #20794D; - --quarto-hl-cf-color: #003B4F; - --quarto-hl-op-color: #5E5E5E; - --quarto-hl-er-color: #AD0000; - --quarto-hl-bn-color: #AD0000; - --quarto-hl-al-color: #AD0000; - --quarto-hl-va-color: #111111; - --quarto-hl-bu-color: inherit; - --quarto-hl-ex-color: inherit; - --quarto-hl-pp-color: #AD0000; - --quarto-hl-in-color: #5E5E5E; - --quarto-hl-vs-color: #20794D; - --quarto-hl-wa-color: #5E5E5E; - --quarto-hl-do-color: #5E5E5E; - --quarto-hl-im-color: #00769E; - --quarto-hl-ch-color: #20794D; - --quarto-hl-dt-color: #AD0000; - --quarto-hl-fl-color: #AD0000; - --quarto-hl-co-color: #5E5E5E; - --quarto-hl-cv-color: #5E5E5E; - --quarto-hl-cn-color: #8f5902; - --quarto-hl-sc-color: #5E5E5E; - --quarto-hl-dv-color: #AD0000; - --quarto-hl-kw-color: #003B4F; -} - -/* other quarto variables */ -:root { - --quarto-font-monospace: "Source Code Pro", monospace; -} - -/* syntax highlight based on Pandoc's rules */ -pre > code.sourceCode > span { - color: #003B4F; -} - -code.sourceCode > span { - color: #003B4F; -} - -div.sourceCode, -div.sourceCode pre.sourceCode { - color: #003B4F; -} - -/* Normal */ -code span { - color: #003B4F; -} - -/* Alert */ -code span.al { - color: #AD0000; - font-style: inherit; -} - -/* Annotation */ -code span.an { - color: #5E5E5E; - font-style: inherit; -} - -/* Attribute */ -code span.at { - color: #657422; - font-style: inherit; -} - -/* BaseN */ -code span.bn { - color: #AD0000; - font-style: inherit; -} - -/* BuiltIn */ -code span.bu { - font-style: inherit; -} - -/* ControlFlow */ -code span.cf { - color: #003B4F; - font-weight: bold; - font-style: inherit; -} - -/* Char */ -code span.ch { - color: #20794D; - font-style: inherit; -} - -/* Constant */ -code span.cn { - color: #8f5902; - font-style: inherit; -} - -/* Comment */ -code span.co { - color: #5E5E5E; - font-style: inherit; -} - -/* CommentVar */ -code span.cv { - color: #5E5E5E; - font-style: italic; -} - -/* Documentation */ -code span.do { - color: #5E5E5E; - font-style: italic; -} - -/* DataType */ -code span.dt { - color: #AD0000; - font-style: inherit; -} - -/* DecVal */ -code span.dv { - color: #AD0000; - font-style: inherit; -} - -/* Error */ -code span.er { - color: #AD0000; - font-style: inherit; -} - -/* Extension */ -code span.ex { - font-style: inherit; -} - -/* Float */ -code span.fl { - color: #AD0000; - font-style: inherit; -} - -/* Function */ -code span.fu { - color: #4758AB; - font-style: inherit; -} - -/* Import */ -code span.im { - color: #00769E; - font-style: inherit; -} - -/* Information */ -code span.in { - color: #5E5E5E; - font-style: inherit; -} - -/* Keyword */ -code span.kw { - color: #003B4F; - font-weight: bold; - font-style: inherit; -} - -/* Operator */ -code span.op { - color: #5E5E5E; - font-style: inherit; -} - -/* Other */ -code span.ot { - color: #003B4F; - font-style: inherit; -} - -/* Preprocessor */ -code span.pp { - color: #AD0000; - font-style: inherit; -} - -/* SpecialChar */ -code span.sc { - color: #5E5E5E; - font-style: inherit; -} - -/* SpecialString */ -code span.ss { - color: #20794D; - font-style: inherit; -} - -/* String */ -code span.st { - color: #20794D; - font-style: inherit; -} - -/* Variable */ -code span.va { - color: #111111; - font-style: inherit; -} - -/* VerbatimString */ -code span.vs { - color: #20794D; - font-style: inherit; -} - -/* Warning */ -code span.wa { - color: #5E5E5E; - font-style: italic; -} - -.prevent-inlining { - content: "