update readme for 3.1.0 release

lambdajack · lambdajack · commit 82eab28db280 · 2022-06-30T15:33:09.000+01:00
diff --git a/README.md b/README.md
@@ -9,18 +9,18 @@
 ## TL;DR give me planet vector tiles! (usage)
 
 1. Have [Docker](https://docs.docker.com/get-docker/) installed.
-2. Download the binary from the [release page](https://github.com/lambdajack/sequentially-generate-planet-mbtiles/releases)
+2. Download the latest binary for your operating system from the [release page](https://github.com/lambdajack/sequentially-generate-planet-mbtiles/releases)
 3. ```bash
-   sudo ./sequentially-generate-planet-mbtiles--v3.1.0-rc4
+   sudo ./sequentially-generate-planet-mbtiles--unix-amd64-v3.1.0
    ```
-4. Rejoice - see acknowledgements below for people to thank.
+4. Rejoice! See acknowledgements below for people to thank.
 
 ## Configuration
 
 ### config.json
 
 ```bash
-sudo ./sequentially-generate-planet-mbtiles--v3.1.0-rc4 -c /path/to/config.json
+sudo ./sequentially-generate-planet-mbtiles--unix-amd64-v3.1.0 -c /path/to/config.json
 ```
 
 ```json
@@ -82,31 +82,31 @@ sudo ./sequentially-generate-planet-mbtiles--v3.1.0-rc4 -c /path/to/config.json
 
 ## Why?
 
-There are some wonderful options out there for generating and serving your own map data and there are many reasons to want to do so. My reason, and the inspiration for this program was cost. It is expensive to use a paid tile server option after less users using it than you might think. The problem is, when trying to host your own, a lot of research has shown me that almost all solutions for self generating tiles for a map server require hugely expensive hardware to even complete (it's not uncommon to see requirements for 64 cores and 128gb RAM!). Indeed the largest I've seen wanted 150gb of the stuff!. For generating the planet that is. If you want a small section of the world, then it is much easier. But I need the planet - so what to do? Generate smaller sections of the world, then combine them.
+There are some wonderful options out there for generating and serving your own map data and there are many reasons to want to do so. My reason, and the inspiration for this program was cost. It is expensive to use a paid tile server option after less users using it than you might think. The problem is, when trying to host your own, a lot of research has shown me that almost all solutions for self generating tiles for a map server require hugely expensive hardware to even complete (it's not uncommon to see requirements for 64 cores and 128 GB RAM!); indeed the largest I've seen wanted 150 GB of the stuff! (for generating the planet that is). If you want a small section of the world, then it is much easier, but I need the planet - so what to do? Generate smaller sections of the world, then combine them.
 
-That's where [sequentially-generate-planet-mbtiles](https://github.com/lambdajack/sequentially-generate-planet-mbtiles) comes in. It downloads the latest osm data, splits it into manageable chunks, generates mbtiles from those chunks and then stitches it all together.
+That's where [sequentially-generate-planet-mbtiles](https://github.com/lambdajack/sequentially-generate-planet-mbtiles) comes in. It downloads the latest osm data (or uses your supplied pbf file), splits it into manageable chunks, generates mbtiles from those chunks and then stitches it all together.
 
 **_This program aims to be a simple set and forget, one liner which gives anyone - a way to get a full-featured and bang up to date set of vector tiles for the entire planet on small hardware._**
 
-It's also designed (work in progress) to be fail safe - meaning that if your hardware (or our software) does crash mid process, you have not lost all your data, and you are able to start again from a point mid-way through.
+It's also designed (work in progress) to be fail safe - meaning that if your hardware (or our software) does crash mid process, you have not lost all your data, and you are able to resume the work where the program left off.
 
-This also uses the openmaptiles mbtiles spec, meaning that when accessing the served tiles you can easily use most of the open source styles available. The default is aimed at using the OSM Bright style. More information on styles can be found below.
+This also uses the openmaptiles mbtiles spec (by default but this can be changed), meaning that when accessing the served tiles you can easily use most of the open source styles available. See our included styles which we can target - you may wish to use those on your front end for a quick setup. More information on styles can be found below.
 
 ## Considerations
 
 1. Hardware usage - this will consume effectively 100% CPU for up to a few days and will also do millions of read/writes from ssd/RAM/CPUcache. While modern hardware and vps' are perfectly capable of handling this, if you are using old hardware, beware that its remaining lifespan may be significantly reduced.
 2. Cost - related to the above, while this program and everything it uses is entirely free and open source - the person's/company's computer you're running it on might charge you electricity/load costs etc. Please check with your provider, how they handle fair use.
 3. Time - your hardware will be unable to do anything much other than run this program while it is running. This is in order to be efficient and is by design. If your hardware is hosting other production software or will be needed for other things in the next few days, be aware that it will perform suboptimally while this is running.
-4. Bandwidth - this will download the entire planet's worth of openstreetmap data directly from OSM. At the time of writing, this is approx. 64GB. **Please note: ** the program will look for a `planet-latest.osm.pbf` file in the `data/pbf` folder. If this is already present, it will skip the download and use this file. If you already have the data you wish to generate mbtiles for, you can place it there to skip the download. This can be useful if you want historical data, or are generating the mbtiles on multiple computers.
-5. Data generation - in order to remain relatively fast on low spec hardware, this program systematically breaks up the OSM data into more manegable chunks before processing. Therefore, expect around 300GB of storage to be used up on completion.
+4. Bandwidth - this will download the entire planet's worth of openstreetmap data directly from OSM. At the time of writing, this is approx. 64 GB. **Please note: ** the program will look for a `planet-latest.osm.pbf` file in the `data/pbf` folder. If this is already present, it will skip the download and use this file. If you already have the data you wish to generate mbtiles for, you can place it there to skip the download. This can be useful if you want historical data, or are generating the mbtiles on multiple computers.
+5. Data generation - in order to remain relatively fast on low spec hardware, this program systematically breaks up the OSM data into more manegable chunks before processing. Therefore, expect around 300 GB of storage to be used up on completion.
 
 ## Requirements
 
 ### Hardware
 
-1. About 300GB clear disk space for the entire planet. Probably an SSD unless you like pain, suffering and the watching the slow creep of old age.
-2. About 4gb of clear RAM (so maybe 6gb if used on a desktop pc). We are working on options in the future for lower RAM requirements.
-3. Time. As above, this has been written to massively streamline the process of getting a planetary vector tile set for the average person who might not have the strongest hardware or the desire to spend £££ on a 64 core 128gb RAM server. Unfortunately, if you cut out the cost, you increase the time. Expect the process to take a couple of days from start to finish on average hardware.
+1. About 300 GB clear disk space for the entire planet. Probably an SSD unless you like pain, suffering and the watching the slow creep of old age.
+2. About 3 GB of clear RAM (so maybe 4/5 GB if used on a desktop pc). We are working on options in the future for lower RAM requirements.
+3. Time. As above, this has been written to massively streamline the process of getting a planetary vector tile set for the average person who might not have the strongest hardware or the desire to spend £££ on a 64 core 128 GB RAM server. Unfortunately, if you cut out the cost, you increase the time. Expect the process to take a couple of days from start to finish on small/old hardware.
 
 ### Software
 
@@ -126,29 +126,27 @@ docker run --rm -it -v $(pwd)/data:/data -p 8080:80 maptiler/tileserver-gl
 
 ## Styles
 
-The default output of `sequentially-generate-planet-mbtiles` looks to match with the open source OSM ['Bright'](https://github.com/openmaptiles/osm-bright-gl-style/blob/master/style.json) style.
+The default output of `sequentially-generate-planet-mbtiles` looks to match with the open source OSM ['Bright'](configs/styles/sgpm-bright.json) style.
 
-When accessing your tileserver with something like [MapLibre](https://maplibre.org/maplibre-gl-js-docs/api/) from a front end application, a good place to start would be passing it a copy of the above ['Bright'](https://github.com/openmaptiles/osm-bright-gl-style/blob/master/style.json) style, **making sure to edit the urls to point to the correct places**.
+When accessing your tileserver with something like [MapLibre](https://maplibre.org/maplibre-gl-js-docs/api/) from a front end application, a good place to start would be passing it a copy of the above ['Bright'](configs/styles/sgpm-bright.json) style, **making sure to edit the urls to point to the correct places**.
 
-You can edit the output of `sequentially-generate-planet-mbtiles` by providing a customised process or config file through the config file.
+You can edit the output of `sequentially-generate-planet-mbtiles` by providing a customised process or config file through the config file. We also support using the tileserver-gl ['Basic'](config/styles/sgpm-basic.json) style.
 
 ### Some style considerations
 
 If making your own style or editing an existing one, note that `sequentially-generate-planet-mbtiles` by default will write text to the `name:latin` tag. If your maps are displayed, but missing text, check that your style is looking for `name:latin` and not something else (e.g. simply `name`).
 
 Pay attention to your fonts. The OSM Bright style makes use of Noto Sans variants (bold, italic & regular). If you are using tileserver-gl to serve your tiles, it only comes with the regular variant of Noto Sans (not the bold or the italic); therefore, it may look like text labels are missing since the style won't be able to find the fonts. You should therefore consider editing the style and changing all mentions of the font to use only the regular variant. Alternatively, you could ensure that all fonts necessary are present.
 
-**Further to the above, please find in this repo, a slightly [edited](./configs/styles/sgpm-bright.json) OSM Bright style for use with the default tileserver-gl. Feed this to your MapLibre or similar front end for a pleasent map suitable for most use cases.**
-
 ## FAQ
 
-1. **How long will this take?** Low spec hardware? Whole planet? A few days. Maybe less than 48 hours for 16 CPUs.
-2. **Would I use this if I have powerful hardware?** Maybe. Since the program essentially saves its progress as it goes, even if you have strong hardware, you are reducing the time taken to redo the process in the event of a crash or file corruption. Further, the RAM is what is really saved here so if you have say 32 cores and 64gb RAM, you still would not be able to generate the entire planet by loading it into memory. Additionally, it just saves time configuring everything.
+1. **How long will this take?** We are pleased to anounce that the new version 3 slicing algorithm is seen to be about twice as fast as the one used in version 2. A modernish 8core CPU/16 GB pc should take less than 24 hours (depending on download speed of course).
+2. **Would I use this if I have powerful hardware?** Maybe. Since the program essentially saves its progress as it goes, even if you have strong hardware, you are reducing the time taken to redo the process in the event of a crash or file corruption. Further, the RAM is what is really saved here so if you have say 32 cores and 64 GB RAM, you still would not be able to generate the entire planet by loading it into memory. Additionally, it just saves time configuring everything.
 3. **Why do I have to run part of the program with 'sudo' privileges?** Many docker installations require sudo to be executed. You may not have to execute the program with sudo.
 4. **Do I have to download the entire planet?** At present, yes. Since if you are not downloading the entire planet, there are other tools out there which do a fine job of getting you mbtiles. We are working on being able to generate mbtiles for smaller areas (such as continents which may still not fit into the average computers RAM)
-5. **Does 'low spec' mean I can run it on my toaster?** Maybe, but mostly not. But you can happily run it on you 4core CPU/4gb RAM home pc without too much trouble. Just time.
-6. **Didn't this used to use GeoFabrik?** It did but the plan was always to move away from geofabrik sources for the planet since it felt unnecessary, when the data was already available direct from source. Further, the GeoFabrik data leaves gaps in the ocean and some of their slices require more than 4gb RAM to process in memory. Ultimately, by getting the data from source, we have more control over it.
-7. **Why would I use this over Planetiler?** Planetiler is a fantastic project, however it still requires minimum 32gb RAM to complete the entire planet (0.5x the size of the planet pbf file).
+5. **Does 'low spec' mean I can run it on my toaster?** Maybe, but mostly not. But you can happily run it on you 4core CPU/4 GB RAM home pc without too much trouble. Just time.
+6. **Didn't this used to use GeoFabrik?** It did but the plan was always to move away from geofabrik sources for the planet since it felt unnecessary, when the data was already available direct from source. Further, the GeoFabrik data leaves gaps in the ocean and some of their slices require more than 4 GB RAM to process in memory. Ultimately, by getting the data from source, we have more control over it.
+7. **Why would I use this over Planetiler?** Planetiler is a fantastic project, however it still requires minimum 32 GB RAM to complete the entire planet (0.5x the size of the planet pbf file).
 
 ## Examples
 
@@ -157,19 +155,19 @@ Pay attention to your fonts. The OSM Bright style makes use of Noto Sans variant
 Use all defaults:
 
 ```bash
-sudo ./sequentially-generate-planet-mbtiles--v3.1.0-rc4
+sudo ./sequentially-generate-planet-mbtiles--unix-amd64-v3.1.0
 ```
 
 Providing a config.json:
 
 ```bash
-sudo ./sequentially-generate-planet-mbtiles--v3.1.0-rc4 -c /path/to/config.json
+sudo ./sequentially-generate-planet-mbtiles--unix-amd64-v3.1.0 -c /path/to/config.json
 ```
 
 Use a specific source file, and send the output to a specific place. Target style to match sgpm-bright:
 
 ```bash
-sudo ./sequentially-generate-planet-mbtiles--v3.1.0-rc4 -p /path/to/planet-latest.osm.pbf -o /path/to/output/dir -tp sgpm-bright
+sudo ./sequentially-generate-planet-mbtiles--unix-amd64-v3.1.0 -p /path/to/planet-latest.osm.pbf -o /path/to/output/dir -tp sgpm-bright
 ```
 
 ## Acknowledgements
@@ -190,11 +188,11 @@ Files generated by `sequentially-generate-planet-mbtiles` are subject to the lic
 
 All welcome! Feature request, pull request, bug reports/fixes etc - go for it.
 
-#### Currently working on:
+## Version 2 (old)
+
+See the version 2 [README](https://github.com/lambdajack/sequentially-generate-planet-mbtiles/blob/9ff1bfcba95dc6f3564c1aade8421bdf1a98fe73/README.md).
+
+### Currently working on:
 
-- Even less ram as an option (aiming for less than 1gb used while still retaining the current speed)
-- More configuration options (including allowing greater data separation for better storage management and the option to use a custom osm.pbf file)
-- Improved logging and progress management
-- Add an automatic end to end test which downloads a small pbf file for quick turn around.
 - v4.0.0 milestone - add automatic fetching of pbf files for continents to use instead of the planet.
 - v4.0.0 milestone - ability to download and merge osm updates to existing mbtiles.
