Skip to content

Commit bd789e0

Browse files
amyheatheramyheather
committed
docs(reproduction): add nhs_rap and heather_2025 checklists completed
1 parent c72be85 commit bd789e0

File tree

2 files changed

+88
-0
lines changed

2 files changed

+88
-0
lines changed

‎docs/heather_2025.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Reproducibility recommendations from Heather et al. 2025
2+
3+
As part of the project STARS (Sharing Tools and Artefacts for Reproducible Simulations), a series of computational reproducibility assessments were conducted by Heather et al. 2025 (**TODO: add DOI of pre-print**). From these, several recommendations were shared to support reproducibility of healthcare discrete-event simulation (DES) models. These are copied below. Those marked with a star (⭐) were identified as having the greatest impact in Heather et al. 2025.
4+
5+
## Recommendations to support reproduction
6+
7+
| Recommendation | Completion | Further details |
8+
| - | - | - |
9+
| **Set-up** |
10+
| Share code with an open licence (⭐) | ✅ | `LICENSE` and `LICENSE.md` |
11+
| Link publication to a specific version of the code | ✅ | - |
12+
| List dependencies and versions | ✅ | `renv.lock` and `DESCRIPTION` |
13+
| **Running the model** |
14+
| Provide code for all scenarios and sensitivity analyses (⭐) | ✅ | `analysis.Rmd` |
15+
| Ensure model parameters are correct (⭐) | ✅ | - |
16+
| Control randomness | ✅ | - |
17+
| **Outputs** |
18+
| Include code to calculate all required model outputs (⭐) | ✅ | - |
19+
| Include code to generate the tables, figures, and other reported results (⭐) | ✅ | `analysis.Rmd` |
20+
21+
## Recommendations to support troubleshooting and reuse
22+
23+
| Recommendation | Completion | Further details |
24+
| - | - | - |
25+
| **Design** |
26+
| Separate model code from applications | ✅ | - |
27+
| Avoid hard-coded parameters | ✅ | - |
28+
| Minimise code duplication | ✅ | - |
29+
| **Clarity** |
30+
| Comment sufficiently | ✅ | - |
31+
| Ensure clarity and consistency in the model results tables | ✅ | - |
32+
| Include run instructions | ✅ | - |
33+
| State run times and machine specifications | ✅ | In `README.md` and `.Rmd` files. |
34+
| **Functionality** |
35+
| Optimise model run time | ✅ | Provides option for parallel processing. |
36+
| Save outputs to a file | ✅ | - |
37+
| Avoid excessive output files | ✅ | - |
38+
| Address large file sizes | N/A | - |

‎docs/nhs_rap.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# 'Levels of RAP' Maturity Framework
2+
3+
The following framework has been directly copied from the RAP Community of Practice repository/website: [NHS RAP Levels of RAP Framework](https://nhsdigital.github.io/rap-community-of-practice/introduction_to_RAP/levels_of_RAP/).
4+
5+
This framework is maintained by the NHS RAP Community of Practice and is © 2024 Crown Copyright (NHS England), shared by them under the terms of the [Open Government 3.0 licence](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/).
6+
7+
The specific version of the framework copied below is that from commit [2549256](https://github.com/NHSDigital/rap-community-of-practice/commit/2549256498886d6d7ea4cdb736e2a2864c8bb461) (9th September 2024).
8+
9+
## 🥉 Baseline
10+
11+
RAP fundamentals offering resilience against future change.
12+
13+
| Criteria | Completion | Further details |
14+
| - | - | - |
15+
| Data produced by code in an open-source language (e.g., Python, R, SQL). | ✅ | R |
16+
| Code is version controlled (see [Git basics](https://nhsdigital.github.io/rap-community-of-practice/training_resources/git/introduction-to-git/) and [using Git collaboratively](https://nhsdigital.github.io/rap-community-of-practice/training_resources/git/using-git-collaboratively/) guides). | ✅ | [GitHub](https://github.com/pythonhealthdatascience/rdesrap_stroke) |
17+
| Repository includes a README.md file (or equivalent) that clearly details steps a user must follow to reproduce the code (use [NHS Open Source Policy section on Readmes](https://github.com/nhsx/open-source-policy/blob/main/open-source-policy.md#b-readmes) as a guide). | ✅ | - |
18+
| Code has been [peer reviewed](https://nhsdigital.github.io/rap-community-of-practice/implementing_RAP/workflow/code-review/). | ✅ | Peer reviewed by Tom Monks |
19+
| Code is [published in the open](https://nhsdigital.github.io/rap-community-of-practice/implementing_RAP/publishing_code/how-to-publish-your-code-in-the-open/) and linked to & from accompanying publication (if relevant). | ✅ | - |
20+
21+
## 🥈 Silver
22+
23+
Implementing best practice by following good analytical and software engineering standards.
24+
25+
Meeting all of the above requirements, plus:https://github.com/pythonhealthdatascience/rdesrap_mms
26+
27+
| Criteria | Completion | Further details |
28+
| - | - | - |
29+
| Outputs are produced by code with minimal manual intervention. | ✅ | - |
30+
| Code is well-documented including user guidance, explanation of code structure & methodology and [docstrings](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/python-functions/#documentation) for functions. | ✅ | - |
31+
| Code is well-organised following [standard directory format](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/project-structure-and-packaging/). | ✅ | - |
32+
| [Reusable functions](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/python-functions/) and/or classes are used where appropriate. | ✅ | - |
33+
| Code adheres to agreed coding standards (e.g PEP8, [style guide for Pyspark](https://nhsdigital.github.io/rap-community-of-practice/training_resources/pyspark/pyspark-style-guide/)). | ✅ | - |
34+
| Pipeline includes a testing framework ([unit tests](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/unit-testing/), [back tests](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/backtesting/)). | ✅ | `tests/` contains unit, functional and back tests. |
35+
| Repository includes dependency information (e.g. [requirements.txt](https://pip.pypa.io/en/stable/user_guide/#requirements-files), [PipFile](https://github.com/pypa/pipfile/blob/main/README.rst), [environment.yml](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/virtual-environments/conda/)). | ✅ | `renv` |
36+
| [Logs](https://nhsdigital.github.io/rap-community-of-practice/training_resources/python/logging-and-error-handling/) are automatically recorded by the pipeline to ensure outputs are as expected. | ✅ | - |
37+
| Data is handled and output in a [Tidy data format](https://medium.com/@kimrodrikwa/untidy-data-a90b6e3ebe4c). | ✅ | Meets the requirements of tidy data (each variable forms a column, each observation forms a row, each type of observational unit forms a table). |
38+
39+
## 🥇 Gold
40+
41+
Analysis as a product to further elevate your analytical work and enhance its reusability to the public.
42+
43+
Meeting all of the above requirements, plus:
44+
45+
| Criteria | Completion | Further details |
46+
| - | - | - |
47+
| Code is fully [packaged](https://packaging.python.org/en/latest/). | ✅ | - |
48+
| Repository automatically runs tests etc. via CI/CD or a different integration/deployment tool e.g. [GitHub Actions](https://docs.github.com/en/actions). | ✅ | `.github/workflows/R-CMD-check.yaml` |
49+
| Process runs based on event-based triggers (e.g., new data in database) or on a schedule. | N/A | - |
50+
| Changes to the RAP are clearly signposted. E.g. a changelog in the package, releases etc. (See gov.uk info on [Semantic Versioning](https://github.com/alphagov/govuk-frontend/blob/main/docs/contributing/versioning.md)). | ✅ | `NEWS.md` and GitHub releases. |

0 commit comments

Comments
 (0)