Cache for previously coded addresses

[sakrejda-umich]

Description

Without changing any user-facing syntax, it would be helpful to make it possible to avoid re-querying the remote API's based on a local cache file.

Motivation

During exploratory data analysis on some larger datasets it's common to surface issues that can be corrected prior to geo-coding but it's helpful to just see which addresses fail in order to identify those issues. Another user has mentioned that this would be desirable previously for what it's worth but I think it's a different and simpler feature than the feature their comment is under. (#39 (comment))

Your Contribution

I can implement a first version of the feature based on hashing

1. the packaged unique inputs as they will be sent to the geocoding service;
2. the identity of the geocoding service; and
3. the arguments that will be sent to the geocoding service

I think it would make sense to merged the cached results transparently. I think it would also make sense to add a function that the user can apply to find a hash for a given record and also a function that allows specific hashes to be dropped from the cache in case the user wants to force re-coding. The (opt-in) cache can likely just be a tibble or set of tibbles saved to a .rds file.

I've implemented some hacky versions of this locally but opened the issue to find out if you have opinions about how something like that should be implemented within the package.

[jessecambon]

Hi @sakrejda-umich, I think this is a good idea but haven't looked into it. Where would the results be cached? If there is another package that caches data it could be useful to look at how they do it.

This might be worth looking into: https://github.com/HenrikBengtsson/R.cache. Or perhaps just using memoise directly? https://memoise.r-lib.org/index.html

[ottothecow]

This is an enhancement I've thought about a lot. As noted in my comment on #39, usually I just implement the behavior on my end when I think it will be useful. The two main reasons it is useful are: 1. Avoiding time/money costs of geocoding many addresses and 2. ensuring reproducibility since geocoding API results can change at any time.

IMHO, the R caching libraries don't really work for this use case, especially if you are doing address in batches (because your actual funciton/API calls will rarely be identical if you are still tweaking the data). Perhaps "Cache" isn't quite the right term as these results are not ephemeral--we want a persistent local lookup that won't change.

I've considered trying to either build it into tidygeocoder, or to build a wrapper around tidygeocoder that does it--just haven't had the time/occasion to make it happen and think through the details of what it would take to make it robust across the full functionality of different geocoders.

Some thoughts that I've had based on my personal solutions are:

1. Should only return exact matches. Same address fields, same geocoder, same options. Any small difference and it gets re-run and the new result added to the cache.
2. Storing as a tibble/dataframe in a .RDS works fine as the cache. I believe this is roughly how tidycensus handles caching ACS data (although their data is far less dynamic than geocoding results). Could implement with SQLite or DuckDB or similar, but I don't think it is worth the extra dependencies.
3. Cache location can be set as an option like an API key. Could also place to a sensible default or use rappdirs::user_cache_dir, but I think it might be worth requiring a location to be explicitly set as these files can grow large with many addreses.
4. I don't have a strong preference for hashing vs just storing some object/columns that contain the relevant information to check if a query matches. Hashing likely more efficient, but not sure the overhead from standard joins is problematic here.
5. Timestamps (and API version type responses) should be captured and retained so you can tell how stale a record is.
6. In a user-facing implementation, there needs to be some way to flush old records and/or flag an input to supercede the cached result. In my DIY solutions, I just do this manually by manipulating the cache dataframe, but that's clearly dangerous. So it begs for some functionality that places guardrails around your cache but still lets you do things like "flush results from before 2/14/2024" or "redo the geocoding for these 10 addresses"
7. Unclear how to handle using multiple geocoders. All one cache? Separate cache for each? In my own work, I usually do a separate one for each (since the variables/names change) which works OK because I am usually cascading through them--try everything in geocoder A, then pass only low-quality matches to geocoder B, then to C, etc.--but with a more clever data structure, these could be in a single location.

I don't think this is a super important enhancement, but it is probably a good idea for people who are looking reproducibility and/or cost control, e.g. academic researchers.