Add blog post on Epiverse community contribution #395
Conversation
✅ Deploy Preview for tourmaline-marshmallow-241b40 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
sbfnk
left a comment
sbfnk
left a comment
There was a problem hiding this comment.
Thanks for putting this together - some initial comments.
|
|
||
| These issues around software sustainability and the academic structures that hinder software longevity were raised by @kucharskiCOVID19ResponseIllustrates2020 and were one of the leading reasons for the [Epiverse-TRACE initiative](https://epiverse-trace.github.io/). Alongside the developing novel software (R packages), Epiverse also has a commitment to support the community of package developers in epidemiology and outbreak analytics. | ||
|
|
||
| This blog post highlights some recent work by Epiverse software engineers to collaborate on research software, or researchware, to help develop an R package that was initially written in the early days of the COVID-19 pandemic (January 2020 - May 2020) and built on code written for the 2014-2016 West Africa Ebola outbreak. Code that has provided insights into ring vaccination [@kucharskiEffectivenessRingVaccination2016] and isolation and contact tracing effectiveness [@hellewellFeasibilityControllingCOVID192020]. Thus it could be of great help in future infectious disease outbreaks, but has been dormant for the past few years. |
There was a problem hiding this comment.
The text later focuses on improvements so I'd use that here:
| This blog post highlights some recent work by Epiverse software engineers to collaborate on research software, or researchware, to help develop an R package that was initially written in the early days of the COVID-19 pandemic (January 2020 - May 2020) and built on code written for the 2014-2016 West Africa Ebola outbreak. Code that has provided insights into ring vaccination [@kucharskiEffectivenessRingVaccination2016] and isolation and contact tracing effectiveness [@hellewellFeasibilityControllingCOVID192020]. Thus it could be of great help in future infectious disease outbreaks, but has been dormant for the past few years. | |
| This blog post highlights some recent work by Epiverse software engineers to collaborate on research software, or researchware, to help improve an R package that was initially written in the early days of the COVID-19 pandemic (January 2020 - May 2020) and built on code written for the 2014-2016 West Africa Ebola outbreak. Code that has provided insights into ring vaccination [@kucharskiEffectivenessRingVaccination2016] and isolation and contact tracing effectiveness [@hellewellFeasibilityControllingCOVID192020]. Thus it could be of great help in future infectious disease outbreaks, but has been dormant for the past few years. |
There was a problem hiding this comment.
I went with "improve", resolved in 47d6936. I couldn't commit as part of the batch of other suggestions due to a conflict with another suggested change on the same line.
|
|
||
| ## The R package | ||
|
|
||
| The R package in question is [{ringbp}](https://github.com/epiforecasts/ringbp). The package has two pieces of functionality: 1) to simulate an infectious disease outbreak using a branching process model with non-pharmaceutical interventions; and 2) to calculate the proportion of simulated outbreaks that are contained (i.e. do not cause a large sustained human-to-human epidemic). The utility of the package's general model framework has been shown by serving as a template for other epidemiological research such as [post-exposure prophylaxis](https://sophiemeakin.github.io/pepbp/). |
There was a problem hiding this comment.
maybe also add other implementations based on ringbp, e.g. https://github.com/lgs85/covidhm and https://github.com/timcdlucas/ringbp
There was a problem hiding this comment.
I've added a mention to both of these in 2cc2e67. I wasn't sure whether to link to the GitHub repo or the paper/pre-print that used them so decided to do both. I couldn't find a paper using {pepbp} but if there is one please point to it and I'll add a citation.
|
|
||
| ## Epiverse contribution | ||
|
|
||
| In the recent months Epiverse has collaborated with {ringbp} developers Seb Funk and Carl Pearson, based at the London School of Hygiene and Tropical Medicine and University of North Carolina, respectively, to try and improve the R package, both internally and from the user-experience. The following sections will give brief summaries of some of the collaborative developments. |
There was a problem hiding this comment.
We should probably somehow clarify that I'm also part of epiverse (and perhaps that that the package existed before), otherwise it may sound like an external collaboration which strictly speaking it wasn't.
There was a problem hiding this comment.
agree; also useful to emphasize i'm not epiverse, since (as I understand it) part of the epiverse remit is making these packages more external-collab/contrib-friendly?
There was a problem hiding this comment.
@pearsonca I've phrased your role with Epiverse as external collaborator, if you'd like this rephrasing let me know.
|
|
||
| Users can now specify the proportion of presymptomatic transmission rather than having to understand the skew normal parameterisation used by the simulation model, making it easier to get started with the package for new users. | ||
|
|
||
| Lastly on user-facing changes, the naming and style of function arguments has been standardised for consistent use of [snakecase](https://en.wikipedia.org/wiki/Snake_case) style and abbreviations. |
There was a problem hiding this comment.
mention that this was previously a mix of stuff? (perhaps in "the problem" above)
There was a problem hiding this comment.
The problem section mentions code style and here we mention standardisation so I would say it's implicit that there was a mix of styles previously. I'll leave as is, as I don't want the details to obfuscate the overarching message, I guess that if anyone is interested they can see the package changelog or version history (linked at the end of this post).
There was a problem hiding this comment.
Can't the "Problem" section have various sections/paragraphs that describe the problems so that they're all in one place? Problems could be introduced as topic sentences in various paragraphs for each kind of problem.
|
|
||
| ### Miscellaneous | ||
|
|
||
| There are various other changes in {ringbp} from our work. Examples include: input checking, not specifying erroneous function defaults, updating the package website, and functions that return `data.table` objects no longer [return silently](https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html#sec:why-do-i-have-to-type-dt-sometimes-twice-after-using-to-print-the-result-to-console). Mentioned in the introduction, model performance has been incrementally improved, but we've not focused on this aspect, and the package will benefit from time spent focusing on this in the future; especially if the set and complexity of non-pharmaceutical interventions in the model expands. |
There was a problem hiding this comment.
| There are various other changes in {ringbp} from our work. Examples include: input checking, not specifying erroneous function defaults, updating the package website, and functions that return `data.table` objects no longer [return silently](https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html#sec:why-do-i-have-to-type-dt-sometimes-twice-after-using-to-print-the-result-to-console). Mentioned in the introduction, model performance has been incrementally improved, but we've not focused on this aspect, and the package will benefit from time spent focusing on this in the future; especially if the set and complexity of non-pharmaceutical interventions in the model expands. | |
| There are various other changes in {ringbp} from our work. Examples include: input checking, not specifying erroneous function defaults, updating the package website, and functions that return `data.table` objects no longer [returning silently](https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html#sec:why-do-i-have-to-type-dt-sometimes-twice-after-using-to-print-the-result-to-console). Mentioned in the introduction, model performance has been incrementally improved, but we've not focused on this aspect, and the package will benefit from time spent focusing on this in the future; especially if the set and complexity of non-pharmaceutical interventions in the model expands. |
There was a problem hiding this comment.
"incrementally" seems a bit of an undersell - I recall getting some order of magnitude reductions in run times. That no longer hold?
I agree the point here isn't to do a systematic optimization comparison, but if there's a big gain, we should be able to brag a bit about the improvement.
There was a problem hiding this comment.
I've just run a rough benchmarking comparison and the newest version of the package (min-gen-time branch on my fork) has approximately the same (if not marginally slower) runtime than the version before I started working on the package (using package version at dbf85c1). The newer version has smaller memory allocation. Therefore I'll keep the current wording which is ambiguous about "performance".
There was a problem hiding this comment.
hmm! i guess things fell off w/ some flexibility introductions or when incorporating corrections that entail drawing more samples.
|
|
||
| The {ringbp} R package implements a simple but informative model for infectious disease transmission and interventions. When written it included many well-developed aspects, but the time constraints of real-time outbreak response meant several improvements were possible. | ||
|
|
||
| Epiverse-TRACE has the opportunity to not only develop new tooling for pandemic preparedness and response, but to contribute to the ecosystem of open-source software in infectious disease epidemiology. We hope that by covering the collaborative developments of {ringbp}, it can illustrate the benefits of bringing software up to date with best practices, and make tools available, accessible and robust when a new epidemic or pandemic occurs, in turn hopefully removing the need for redeveloping similar software in the future. |
There was a problem hiding this comment.
Do you think there is scope for perhaps a reflection on sustainability / maintenance, which you mentioned extensively in the motivating intro? How do we ensure this is not again abandoned for 5 years (assuming it would be a problem if it was)?
Also perhaps there is scope for a final statement that we're hoping that this will make the package more useful for others and will re-visit the analysis for the paper (with ref to the cmmid ringbp repo) in the first instance?
There was a problem hiding this comment.
On the first item - maybe also lean into all the other contributor-friendliness-items you've added (contributor statement, code of contact, making issue templates, tag clean up, etc) [or will add prior to releasing blog]?
And plan for CRAN release?
There was a problem hiding this comment.
Do you think there is scope for perhaps a reflection on sustainability / maintenance, which you mentioned extensively in the motivating intro? How do we ensure this is not again abandoned for 5 years (assuming it would be a problem if it was)?
I did consider this but initially opted again as I don't think it would be a positive message. Given the current lifecycle stage of Epiverse-TRACE I'm not sure we can make any grand statements about long term improvements in the landscape for OSS for research. Specifically on "How to do we ensure this is not again abandoned for 5 years", I think this is outside the scope of the post and I'm not sure I have an adequate answer. If you disagree, or there is something specific you'd like the post to cover please suggest and I'd be happy to incorporate.
On the first item - maybe also lean into all the other contributor-friendliness-items you've added (contributor statement, code of contact, making issue templates, tag clean up, etc) [or will add prior to releasing blog]?
We could add this to the post but it's thus far not been added to the {ringbp} repository so would be similar to the Testing section of the post which requires some more work on the package before publishing. IIRC there hasn't been any discussion of code of conduct, issue or PR templates; the contributing document has been raised in epiforecasts/ringbp#109.
And plan for CRAN release?
This has been raised in epiforecasts/ringbp#128 and is the plan. It would be nice to publish this post in tandem with the {ringbp} release.
There was a problem hiding this comment.
I think there's a reasonably-but-not-overly optimistic statement here - that the flexibility, refactoring, expanded documentation, etc make it easier for other people to reuse it, which is the gateway for external contributors to engage with the project. That doesn't assure resources on it, but enables them.
I'd also argue that folding it into a larger effort (tho I guess Epiverse-TRACE is wrapping?), instead of just an isolated package, improves awareness.
There was a problem hiding this comment.
Thanks, that's a good point that I'd largely overlooked. I've added a paragraph to the Conclusion to hopefully address this point in 6743d36. Let me know if you think it needs reworking.
tho I guess Epiverse-TRACE is wrapping?
The project is moving into Phase II with a focus on implementation with training and maintenance of tools, so will have a reduced software engineering capacity.
|
|
||
| Software that is developed for research or by researchers can be difficult to maintain given the incentive and funding structures in academia. This remains true for epidemiology, with a large volume of software written during the COVID-19 pandemic, much of which is now abandonware[^1]. This does not mean that the software developed to understand the COVID-19 pandemic was bad or does not have utility in understanding future epidemics and pandemics, but just that the capacity to maintain and further develop these tools is not available now the pandemic is no [longer considered an acute public health emergency](https://www.who.int/news/item/05-05-2023-statement-on-the-fifteenth-meeting-of-the-international-health-regulations-(2005)-emergency-committee-regarding-the-coronavirus-disease-(covid-19)-pandemic). | ||
|
|
||
| These issues around software sustainability and the academic structures that hinder software longevity were raised by @kucharskiCOVID19ResponseIllustrates2020 and were one of the leading reasons for the [Epiverse-TRACE initiative](https://epiverse-trace.github.io/). Alongside the developing novel software (R packages), Epiverse also has a commitment to support the community of package developers in epidemiology and outbreak analytics. |
There was a problem hiding this comment.
Last sentence bothers me mildly; maybe more like One of the key Epiverse objectives is "[pull quote from proposal]".? Not sure what that pull quote looks like, but the current wording here is a bit awkward. I also don't think its necessary to invoke that y'all also mint new software.
There was a problem hiding this comment.
I've gone through some of the early documentation for the project proposal and couldn't find any information that would fit into this context. Related is the mention of the CSCCE Community Participation Model.
I also don't think its necessary to invoke that y'all also mint new software.
I wanted to phrase it this way as I didn't want it come across as contributing to other software is our primary activity. (Added bonus for advertising our tools).
|
|
||
| These issues around software sustainability and the academic structures that hinder software longevity were raised by @kucharskiCOVID19ResponseIllustrates2020 and were one of the leading reasons for the [Epiverse-TRACE initiative](https://epiverse-trace.github.io/). Alongside the developing novel software (R packages), Epiverse also has a commitment to support the community of package developers in epidemiology and outbreak analytics. | ||
|
|
||
| This blog post highlights some recent work by Epiverse software engineers to collaborate on research software, or researchware, to help develop an R package that was initially written in the early days of the COVID-19 pandemic (January 2020 - May 2020) and built on code written for the 2014-2016 West Africa Ebola outbreak. Code that has provided insights into ring vaccination [@kucharskiEffectivenessRingVaccination2016] and isolation and contact tracing effectiveness [@hellewellFeasibilityControllingCOVID192020]. Thus it could be of great help in future infectious disease outbreaks, but has been dormant for the past few years. |
|
|
||
| ## Epiverse contribution | ||
|
|
||
| In the recent months Epiverse has collaborated with {ringbp} developers Seb Funk and Carl Pearson, based at the London School of Hygiene and Tropical Medicine and University of North Carolina, respectively, to try and improve the R package, both internally and from the user-experience. The following sections will give brief summaries of some of the collaborative developments. |
There was a problem hiding this comment.
agree; also useful to emphasize i'm not epiverse, since (as I understand it) part of the epiverse remit is making these packages more external-collab/contrib-friendly?
|
|
||
| ### User interface | ||
|
|
||
| The user experience (API) of the package has been refactored. The main simulation function `scenario_sim()` remains, but its arguments have been modularised to better group model parameters and control arguments. This also makes the package easier to develop further without necessarily introducing many breaking changes and prevents the number of top-level function arguments from expanding. |
|
|
||
| ### Miscellaneous | ||
|
|
||
| There are various other changes in {ringbp} from our work. Examples include: input checking, not specifying erroneous function defaults, updating the package website, and functions that return `data.table` objects no longer [return silently](https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html#sec:why-do-i-have-to-type-dt-sometimes-twice-after-using-to-print-the-result-to-console). Mentioned in the introduction, model performance has been incrementally improved, but we've not focused on this aspect, and the package will benefit from time spent focusing on this in the future; especially if the set and complexity of non-pharmaceutical interventions in the model expands. |
There was a problem hiding this comment.
"incrementally" seems a bit of an undersell - I recall getting some order of magnitude reductions in run times. That no longer hold?
I agree the point here isn't to do a systematic optimization comparison, but if there's a big gain, we should be able to brag a bit about the improvement.
|
|
||
| The {ringbp} R package implements a simple but informative model for infectious disease transmission and interventions. When written it included many well-developed aspects, but the time constraints of real-time outbreak response meant several improvements were possible. | ||
|
|
||
| Epiverse-TRACE has the opportunity to not only develop new tooling for pandemic preparedness and response, but to contribute to the ecosystem of open-source software in infectious disease epidemiology. We hope that by covering the collaborative developments of {ringbp}, it can illustrate the benefits of bringing software up to date with best practices, and make tools available, accessible and robust when a new epidemic or pandemic occurs, in turn hopefully removing the need for redeveloping similar software in the future. |
There was a problem hiding this comment.
On the first item - maybe also lean into all the other contributor-friendliness-items you've added (contributor statement, code of contact, making issue templates, tag clean up, etc) [or will add prior to releasing blog]?
And plan for CRAN release?
chartgerink
left a comment
chartgerink
left a comment
There was a problem hiding this comment.
Happy to schedule this whenever 😊 Thanks @joshwlambert for the blog!
|
Thanks all for your feedback. I've been on leave for the past couple of weeks but will now work through the comments. |
Co-authored-by: Sebastian Funk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: Sebastian Funk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: Carl A. B. Pearson <pearsonca@users.noreply.github.com>
|
@chartgerink can we schedule this to be merged on 25th August? (IIRC we merge posts on Mondays). This should give enough time from now for anyone to give comments and for me to address them. |
|
|
||
| These issues around software sustainability and the academic structures that hinder software longevity were raised by @kucharskiCOVID19ResponseIllustrates2020 and were one of the leading reasons for the [Epiverse-TRACE initiative](https://epiverse-trace.github.io/). Alongside the developing novel software (R packages), Epiverse also has a commitment to support the community of package developers in epidemiology and outbreak analytics. The initiative also tries to improve community collaboration and contribution friendliness of open-source software. | ||
|
|
||
| This blog post highlights some recent work by Epiverse software engineers to collaborate on research software, or researchware, to help improve an R package that was initially written in the early days of the COVID-19 pandemic (January 2020 - May 2020) to assess the effectiveness of isolation and contact tracing effectiveness [@hellewellFeasibilityControllingCOVID192020]. It built on code written for the 2014-2016 West Africa Ebola outbreak to provide insights into ring vaccination [@kucharskiEffectivenessRingVaccination2016]. These applications and the general nature of the questions the package addresses suggest that it could be of great help in future infectious disease outbreaks, but has lacked developer resources without pandemic-related priorities. |
There was a problem hiding this comment.
I think it's fair to mention that the work that package was used to develop received a lot of attention, i,e., citations, and the package was repurposed a few times for other high impact work, i,e., covidhm and others.
There was a problem hiding this comment.
I see this is later mentioned in the next section. I wonder if the two sections can be combined then to make it easier to be sold on the importance of this exercise.
|
✅ Merge Schedule |
|
Thanks for the comments. I've delayed the scheduled merge to make sure I have time to incorporate all the feedback and account for the UK bank holiday. |
|
Editing the merge schedule command above didn't update the merge schedule correctly, so this PR was accidentally merged on the original schedule date (25th), I've made the last few edits from comments in PR #398. |
This PR adds a blog post covering the collaborative work of Epiverse in the wide epi tools ecosystem, focusing on the {ringbp} package. It highlights why collaborative development of open-source research software is important, why {ringbp} was worked on, and gives some of the technical details of the various developments.
Tagging @pearsonca and @sbfnk as collaborators on {ringbp}.
Also assigning members of the Epiverse team to review the blog post before merging.
One question that came up when discussing this post was whether we should name the developer(s) of {ringbp}, and if so whether we name the lead developer or all authors listed in the package, and secondly whether we should link to {pepbp} or cite any research that uses {pepbp}, or mention the developer(s) of {pepbp}. Please can people let me know if they have a preference?
The post is still a draft, and depends on merging a few outstanding developments into the
mainbranch in {ringbp}. The Testing section still needs to be written once regression testing is added to {ringbp}, and the Bug fixes section includes some fixes that are not yet merged into themainbranch on {ringbp}. These are a blocker to resolve before this post can be published.RRight before merging:
datefield has been updatedblueprintsto link to this post_freeze/folder is up-to-date/schedule 2025-08-25