Would you like potential fixes submitted as a Pull request?

[colin0brass]

(UPDATE 15-Aug-2025: correcting my note of gedcom-to-visualmap code version: using tag v0.2.6.0, not v0.2.5.4 as I initially wrote)

Hi @D-Jeffrey , thank you very much for developing and sharing this program.

I had done my own (unpublished) GEDCOM to KML geocoding (with location caching) and visualisation application too, but you have many more features, including the rather nice heatmap visualisation.

It took me a couple of false starts with your program before I got it working for me on Mac. I first tried unsuccessfully in 2022, and then came back to it in 2025, still failed (with current v0.2.6.0), but decided to try harder and made a few minor code changes to resolve some run-time errors.

I can imagine you might well have made further changes since your current latest GitHub version (v0.2.6.0) released 30-Mar-2025. Would you like potential fixes submitted as a Pull request? Or would that just get complicated vs your latest local code-base?

Thinking about it, I suspect there could be a few options including perhaps:

1. I just keep my own local edits, or even fork your code
2. I give you details of my edits, and you check and potentially publish update on GitHub
3. You publish your latest version on GitHub, I update to that, and submit a Pull request with any remaining proposed fixes

(I only needed to change a line or two in each of about 4 python files, plus adding an "init.py" file to help recognise gedcom as a package, so it wouldn't be hard to communicate the changes either way)

Please could you help advise if you would take any such suggestions, and if so how you would prefer?

Regards
Colin

[D-Jeffrey]

Hey Colin,

Thanks for reaching out. I would welcome your changes and make it work properly on a MAC. I a few bug fixes in my work to push up shortly. I'm not a Mac user so I was only guessing how make it OS independent. I have been trying to make the KMl and HTMl a little more feature rich. I struggle with git a bit so I'm not sure I can make a pull/merge work on the first try. (My regular job is to lead IT Ops guys, but I miss development so I do this.)

I'm away on holidays for a week but will followup after that.
So let's figure this out together. What you got?

Darren

[colin0brass]

Thanks Darren. I'll start by describing the issues I have seen, and my current minor fixes, and you could either look to roll those into your near-term bug fixes and push from your side, or if they remain in your next release then I could submit a pull request and/or further details.

My comments below are based on tag v0.2.6.0 as downloaded from GitHub. Showing version:

git log -1
commit 2edd9f69c72db31473a832937d1a2b60ca9650a6 (HEAD -> main, tag: v0.2.6.0, origin/main, origin/HEAD)
Author: D Jeffrey <44452180+D-Jeffrey@users.noreply.github.com>
Date:   Sat Mar 29 21:32:09 2025 -0600

In summary, relating to running GUI version as python gv.py:

• Package "gedcom" failed to load, due to no "init.py" file found
• App launch failed due to undefined GEDCOMinput (file), causing exception in "os.path.split"
• A few mis-match/incompatibilities in option names used in different places
• Libreoffice executable file name needed changing
• "Draw Update" button never enabled (always coloured red) due to copy/paste of BTNUpdate vs BTNLoad

What I started looking into, but haven't yet figured out, is that the HTML options for individually enabling/disabling birth/marriage/death types of markers does not seem to be working. The overall "Markers" enable/disable works fine, but not "Marker for when <Born/Died>".

In more detail: (line numbers might vary slightly where there is more than one edit in the same file)

gedcom package:
I added "init.py" file as gedcom-to-map/gedcom/__init__.py to help python recognise gedcom as a package
(I could imagine you might have that file already yourself but not "git add"-ed it to include in git & GitHub)

File: gedcomVisualGUI.py; Line 278
Added default file name for case where GEDCOMinput is not defined and "os.path.split" can't be called
filen = "gedcomfile.ged"

File: gedcomVisualGUI.py; Line 281
Inside if panel and panel.g0, changed undefined infile detection from '' to None to match initialisation; from:
if infile is != '':
To:
if infile is not None:

File: gedcomVisualGUI.py; Line 1228
In class VisualMapPanel( ... ):
In def OnMyTimer( ... ):; added handling of undefined GEDCOMinput in os.path.split; from:
_, filen = os.path.split(self.gO.get('GEDCOMinput'))
To:
_, filen = os.path.split(self.gO.get('GEDCOMinput') or "")

File: gedcomVisualGUI.py; Line 1235
In class VisualMapPanel( ... ):
In def OnMyTimer( ... ):; corrected what I interpreted was a copy/paste unintended duplication; from:
if not self.id.BTNLoad.IsEnabled():
To:
if not self.id.BTNUpdate.IsEnabled():

File: gedcomVisualGUI.py; Line 1485
In class VisualMapPanel( ... ):
In runCMDfile( ... ); corrected reference to cmdline; from:
if g0p.cmdline.startswith('http'):
To:
if cmdline.startswith('http'):

File: gedcom-to-map.py; Line: 77
In function call for: myGeoOptions.setmarkers( ... )
Changed parameter name from:
AntPath = arg_parse.args.antpath,
To:
UseAntPath = arg_parse.args.antpath,
(to be compatible with use in file: render/foliumExp.py; choice of which way around to make this consistent)

File: gedcomoptions.py; Line 93 and perhaps also 96
In class gvOptions; def __init__( ... )
Changed application executable name for libreoffice from:
self.CSVcmdline = "libreoffice --calc $n"
To:
self.CSVcmdline = "soffice --calc $n"
(from an online search, it appears that 'soffice' is the normal application executable name for Mac & Linux, so I interpret not a local issue on my computer setup)

File: gedcomvisual.py; Line 140
In function: def ParseAndGPS( ... )
Added more protection for incomplete options, from:
gOp.UpdateBackgroundEvent.updategrid = True
To:

        if hasattr(gOp, "UpdateBackgroundEvent") and hasattr(gOp.UpdateBackgroundEvent, "updategrid"):
            gOp.UpdateBackgroundEvent.updategrid = True

[D-Jeffrey]

Seems very minimal. Will be easy to adjust when I get back home.

I will also see if I can get proper dev/push setup and working as well if you have future tweaks.

[D-Jeffrey]

Okay, so I have folded all your changes into the next update. It includes work on the People Dialog as well as work on Soundex matching for the HTML family relations. I will publish it (2.6.2) tomorrow after doing some more testing on my own code updates. (You found a lot of sloppy bugs).

[colin0brass]

Thanks Darren.

I have just updated to v0.2.6.3 and it works fine for me on Mac now. Great.

A few quick thoughts and observations (I'll try to be more directly helpful sometime rather than just a commentator, but in the interests of time I'll start with this).

Input File:

• I was about to say 'it would be nice to add a button on the GUI to browse for input file', but then found that the "Input file:" text on the GUI is already a clickable button. I hadn't spotted that! I wonder if this could be made to look more obviously like a button to help make this more obvious?

HTML Options:
The marker type enable/disable check boxes don't yet seem to work (if I'm correctly interpreting what they should do)

• "Markers" overall enable/disable works fine
• "Marker for when Born" and "Marker for when Died" don't appear to change anything for me
• (for results type: HTML; Group by "Last Name"; Map Style CartoDB.Voyager; using action "Draw Update" and then externally opening resulting html in web browser)

KML Options:
These are less extensive than the HTML options (currently only enable/disable options for 'native', 'born', 'death'), but seem to be working.
Something that I had added in own attempted GEDCOM to KML program was hyperlinks in the pop-up person/point descriptions boxes; e.g. extracts of my own code:

description = point.description
description += 'Father: <a href=#{};balloonFlyto>{}</a><br>'.format(father_id, self.people[current.father].name)
description += 'Mother: <a href=#{};balloonFlyto>{}</a><br>'.format(mother_id, self.people[current.mother].name)
# & for each child:
description += '<a href=#{};balloonFlyto>{}</a> '.format(child_id, self.people[child].name)

If I recall correctly, it was important to set the HREF link to a PLACEMARK ID, since Google Earth didn't appear to be able to hyperlink / fly-to a POINT, only a PLACEMARK. I spent a while battling this and finding explanations that it couldn't be done, but this compromise/implementation did work for me in the end.

I also used TIMESTAMP in the KML PLACEMARK, which then enabled the "time slider" in Google Earth (Pro), to be able to filter and animate people and places through the time period covered by the KML file.

Example extract of KML showing an HREF in the first PLACEMARK (id="7076"), which flies to the second PLACEMARK (id="7079"):

        <Placemark id="7076">
            <name>Thomas Fowler</name>
            <description>Birth 1820&lt;br&gt;St Botolph, Aldersgate, England&lt;br&gt;Father: &lt;a href=#7079;balloonFlyto&gt;William Fowler&lt;/a&gt;&lt;br&gt;Mother: &lt;a href=#6933;balloonFlyto&gt;Jane Billinge&lt;/a&gt;&lt;br&gt;Children: &lt;a href=#7073;balloonFlyto&gt;Susan Emily Fowler&lt;/a&gt; </description>
            <TimeStamp id="7077">
                <when>1820</when>
            </TimeStamp>
            <Point id="7075">
                <coordinates>-0.0972227,51.5170018,0.0</coordinates>
            </Point>
        </Placemark>
        <Placemark id="7079">
            <name>William Fowler</name>
            <description>Birth 1780&lt;br&gt;Staffordshire, England&lt;br&gt;Father: &lt;a href=#7082;balloonFlyto&gt;william Fowler&lt;/a&gt;&lt;br&gt;Mother: &lt;a href=#6883;balloonFlyto&gt;ann&lt;/a&gt;&lt;br&gt;Children: &lt;a href=#7076;balloonFlyto&gt;Thomas Fowler&lt;/a&gt; </description>
            <TimeStamp id="7080">
                <when>1780</when>
            </TimeStamp>
            <Point id="7078">
                <coordinates>-1.866700,52.811050,0.0</coordinates>
            </Point>
        </Placemark>

[D-Jeffrey]

Interesting experience on HTML. I will look to fold in the KML tag format that you suggested. It seems most people are interested in the KML version whereas I have spent all of my time working on HTML version.

Make sure you uncheck the Use GPS cache only options so it takes you addresses and looking them up to get a set of gps values.

Are you GPS points embedded in your GED file? See this. If they are then that would explain why it works in KML mode and not HTMl. Unchecking Marker should put it in Cluster mode (See end example is ReadMe). I'm not sure if the HTML will accept GPS from GED.

The markers appear once there are Geocodw values for born/death/CHR/IMMI and a few more events.

If you right-click (not sure what that is on a MAC - feedback), the. It will bring up the details on a person and that should show you what address and other location events it has on your people.

In the readme near the bottom are examples of what the HTMl experience should be.

In the table of people if the values are (None, None) then it is not resolving addresses. I have feed a lot of different GED sources in there so I wonder if the JavaScript is getting scrambled somehow.

You can use the Options-> Options Setup to set the DEBUG level for different modules. For HTML it is FoliumExp. That is also where you would change the command line for opening in Spreadsheet. And what KML program to use.

Appreciate the feedback (good, bad or otherwise). I will make the open into buttons.

[colin0brass]

(Thanks Darren. I'll look closer another time ... but before I go I should explain better than I did above ... with the HTML marker options enabled/disable, what I'm seeing is that I don't seem to be able to selectively disable different types of markers: if markers are enabled then all types of markers are generated even if I try to disable some types)

[D-Jeffrey]

Evening Colin,

Ah now I understand... sorry, I may have miss-read your first posts... You are correct, now that I looked thru the code, I see that I lost control of displaying birth and death as controlable points, so born/death checkboxes don't do anything. That need refactoring.

I got some of your suggested formatting into the ballon. I did fix the timespan for lines and and timestamp for points. I never knew about the time spanning function. It starts with everything off in ArcGIS, so I will have to tweak that a bit more.

At the current times the balloon does not iterate the child, but it does show:

Dorothy Potter (b)

Father: Aaron POTTER

Mother: Dorcas MUNSON

Lifespan: 1750 to 1824, related as 01010001

<Placemark id="808">
            <name>Dorothy Potter (b)</name>
            <description>&lt;br&gt;Father: Aaron POTTER&lt;/br&gt;&lt;br&gt;Mother: Dorcas MUNSON&lt;/br&gt;&lt;br&gt;Lifespan: 1750 to 1824, related as 01010001&lt;/br&gt;</description>
            <TimeStamp id="809">
                <when>1750</when>
            </TimeStamp>
            <styleUrl>#2</styleUrl>
            <Point id="807">
                <coordinates>-73.11068609212393,41.694321531292296,0.0</coordinates>
            </Point>
        </Placemark>
        <Placemark id="811">
            <name>Dorothy Potter (b)</name>
            <description>&lt;br&gt;Father: Aaron POTTER&lt;/br&gt;&lt;br&gt;Mother: Dorcas MUNSON&lt;/br&gt;&lt;br&gt;Lifespan: 1750 to 1824, related as 01010001&lt;/br&gt;</description>
            <TimeStamp id="812">
                <when>1824</when>
            </TimeStamp>
            <styleUrl>#6</styleUrl>
            <Point id="810">
                <coordinates>-72.92464897264712,41.30845541523654,0.0</coordinates>
            </Point>
        </Placemark>
        <Placemark id="814">
            <name>Dorothy Potter</name>
            <description>&lt;br&gt;Father: Aaron POTTER&lt;/br&gt;&lt;br&gt;Mother: Dorcas MUNSON&lt;/br&gt;&lt;br&gt;Lifespan: 1750 to 1824, related as 01010001&lt;/br&gt;</description>
            <TimeSpan id="817">
                <begin>1750</begin>
                <end>1824</end>
            </TimeSpan>
            <styleUrl>#815</styleUrl>
            <LineString id="813">
                <extrude>1</extrude>
                <tessellate>1</tessellate>
                <altitudeMode>clampToGround</altitudeMode>
                <coordinates>-73.1102814602354,41.69467304887386,0.0 -72.92500628377937,41.30842491176131,0.0</coordinates>
            </LineString>
        </Placemark>

Your right trying to get the ballonflyto to work, it challenges... I will keep working on it. the problem is once a placemark is created, with simplekml, I'm not sure I can go back and edit the placemarks to update the forward reference to a person who is not in the list yet in the generation of the kml. I will figure out how to hack it.

I have chagned the Input/output to real buttons. Need to work on some styling for it and refactoring. Should have this release ready by next weekend.

One question ... is it better to have two end placemarks and 3rd placemark with a linestring (as above) or combine the points with the linestring in one placemark? The implications of combing them is that you cannot tell which is the birth point and which is the death point.

[colin0brass]

Sounds great.

I'll take a closer look at my code and see if I can firstly remember what I was trying to do, and secondly complete a few things that were pending, to have a more complete reference.

My first gedcom to kml program was about 10 years ago in perl, and then I did a partial re-implementation of it in python about 3 years ago but haven't yet got back to all the features I had in my original perl.

Good question about the KML placemark partitioning. I have separate placemark per point and per line. If my hazy recollection is correct that balloonFlyTo only works (at least in Google Earth Pro) with placemark and not point, then as you say, it sounds like we might have better control and granularity by keeping as separate placemarks. I have a limited knowledge, despite a few bruises, in this area.

For the KML placemark IDs for hyperlinks, in my original perl program I had full control, so was able to set from the GEDCOM person ID values, and add suffix to the ID to distinguish birth/marriage/death placemarks. As you say, in simplekml the ID is a constant (if I interpret correctly). What I do currently is to populate all the people points (in placemarks) first, and then go back through to update the people descriptions to include relationships and hyperlinks.

[colin0brass]

Hi Darren, I have spent a while tidying-up my own old code, trying to make it better organised, consistent and readable.

It is only command-line driven, and has very limited output options of kml, a few summary text (csv) files and one heatmap matrix image (png) summary file.

I have tried something new with my kml output, creating and using folders in the kml for different types of event or line. This seemed the easiest way to selectively enable or disable these features dynamically in Google Earth (Pro).

I suspect I might have picked-up some inspiration from your program a few years ago when I was first trying unsuccessfully to run it, so I apologise if anything looks familiar but unattributed. I would be perfectly happy to update that where appropriate.

I did have it as a private repo, but have opted to make it public now for ease of sharing and discussion.
I would be very interested to know if it works for anyone other than me since I've not shared this code at all before.
Beyond that I don't know how much I want to extend and maintain it, but will see how things go ...
(I'm mainly thinking of it as a source of experiments and ideas)

See online as follows ... please feel free to use/ignore/copy/suggest as you see fit:
https://github.com/colin0brass/gedmap

[D-Jeffrey]

Well, that is a nice object driven code base compared to my old school brute force procedural code base. The code cleaned up very nicedly. There are a number of things you accomplished in there that, I was trying to by my brute for code can really benefit from what you are doing. I like you geocode, yaml. much better than my hard coded approach. I like your location enrichment... create to @lmallez for those based classes that I used for the parsing. Reading over your code, there is a lot of 'opportunity' for me to improve my code.

I really like your idea for people and location summary.

Maybe looking over your kml output and I determine what I'm doing wrong in my balloonFlyto.

Pushing the update of where I am now.

[colin0brass]

Thanks. My summary files were motivated by wanting a way to review all locations used in my tree and improve consistency and formatting to better enable geolocation and mappingSent from my iPhoneOn 30 Aug 2025, at 20:10, D Jeffrey ***@***.***> wrote: Well, that is a nice object driven code base compared to my old school brute force procedural code base. The code cleaned up very nicedly. There are a number of things you accomplished in there that, I was trying to by my brute for code can really benefit from what you are doing. I like you geocode, yaml. much better than my hard coded approach. I like your location enrichment... create to @lmallez for those based classes that I used for the parsing. Reading over your code, there is a lot of 'opportunity' for me to improve my code. I really like your idea for people and location summary. Maybe looking over your kml output and I determine what I'm doing wrong in my balloonFlyto. Pushing the update of where I am now. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[colin0brass]

Testing my program on your nice set of GEDCOM samples I realise that my "adding default country" fall-back when it couldn't identify a recognisable country in the original address field is rather over-simple and far too prone to error.

It seemed like a plausible short-cut when I was wrestling with thousands of people in my tree and tens of thousands of location points, many with inconsistent and incomplete location information, but on a wider stage I can see that it often fails horribly.

(I'm now trying re-running using my non-default "--default_country none" to see how that fares ... hopefully less prone to wild geolocation mistakes)

[D-Jeffrey]

Okay, v0.2.6.4.1 has a FlyTo that works and Children. Still working on styling the balloons

[colin0brass]

Thanks. I've pulled your latest main and it works fine for me. As you say, kml FlyTo is working and children list in pop-up bubbles.

By the way, I notice that children list is currently missing in birth placemarks (in kml).
(children list is present and working fine in death and lifespan placemarks)

I assume you will probably add FlyTo hyperlinks for the Children at some point since it is working so nicely on parents.

[D-Jeffrey]

The thought process on not including the child on the birth marker is they are not born yet. They are born sometime during the life (the line). I was thinking how to put the midlife events into the KML model as it is in the HTML model. I need change around the object structure to include the LifetimeCreator object.

I add the FlyTo on the child and I see now how that is a problem to fly back and forth. (Added to born for now as well).

I keep thinking I have logic issues, with the lines and markers... I know the first person is not drawn like the other I need to fix that. I keep thinking I have other labelling problems... but it is a second instance loaded in Earth or a parent with the same name.

I have experimenting with BalloonStyle Formatting, but I cannot come up with a style which make the balloon large enough to avoid a scroll.

UseBalloonFlyto is configuration manually in the INI file now.

TODO: add marker shapes and colors to INI file.... maybe that should be a YAML file.

Pushing v0.2.6.6 (hmmm that was the wrong version ... committed now :( ).

[colin0brass]

Here is an example of my birth & death countries heatmap matrix view, which I developed firstly as an interesting overview of locations, and then more purposefully while reviewing all my location entries and working to improve consistency and ensure good geolocation.

[D-Jeffrey]

Wow, I really like that. I'm not usually a Plot fan, but that looks really great. Definitely need to include that. I can see what you need to country translation to make that happen.

Makes me think of doing variation of a histogram with age of death and age of child birth.

[colin0brass]

This is what it looks like so far for your royal92 sample gedcom ... though I am still wrestling with injecting some of the address disambiguations into the flow ...:

[colin0brass]

Ok, crafted & fed in a csv to give the alt_addr values for better geocoding, and achieved a smoother flow.
I'm sure it isn't perfect, but I was also experimenting with using an AI assistant application to help with finding and filling the gaps and errors in geolocation.
royal92_cache.csv

[D-Jeffrey]

Are you having to do the manual edits 'manual_correction:'? For example in 'Barnwell Church' did you edit the Long for that or come up with an alternate location name. I have long since gave up on trying to correct the output from looking up something in OpenStreetMap. My plan was to someday use some of the other columns in the CSV cache... Sigh it looks like my 'used; column is all missed up on updates (TODO). I had looked at having a cache for every GED file, but depending how you reuse an address, and fix it, then you might not have the benefit when you do other lookups. As well, when you dump your family from some of the tools, they may generate a random file name, and you end up looking up all the same addresses over again. One suggestion would be to add a column into a single cache file to reference the names of the GED where that cache value was used, if the goal is to keep you a 'workable' list limited to the latest people you added.

I was looking to do a 'fix up phases' for the substitution. For example, sometimes there are useless phases in the address like: "po box" or 'of' or 'part of' then you get into Regex pattern match precedence ('part of' before 'of'). Then there are other subs like
r'co.' : 'county',
r'twp.': 'township',
r'tp.': 'township',
r'tp': 'township'}
There are other placement addresses as word matches which should be replaced like you did but at the state/district/province/town level such as
"Co.": "County",
"Alta": "Alberta",
"Sask" : "Saskatchewan",
"Canada West": "Ontario",
"Upper Canada": "Ontario",
"Lower Canada": "Quebec",
"East Canada": "Quebec"

but then it got into you need to match the whole word, and make sure it is positional correct relative to other names in the address. "town, state, country", but the country is missing a lot. That is why my gpslookup looks broken as it recurses. I had run some many GED files records thru it, that it can force fit addresses for cases that are really a stretch.

I do like how you have been able to dig out the country/continent from your addresses. (Filing they idea.)

Anyway, restructured the GUI and corrected the markers in HTML mode to work properly. I think it makes the experience better.
I found a GED loading bug trying out other GED files (rabbit trail),

On the KML side, the birth death should work. Still thinking about the mid life.

New version push (don't use the tag, I missed 2 files (what a mess).

[colin0brass]

With my own tree I did go through making manual edits, partly I felt forced because Ancestry has some nice mapping but seemed to be very US biased when trying to relocate my tree event locations, many of which were not fully specified to the country level.

So I used a combination of Ancestry, my own scripting, and Family Tree Maker app ... to go through ensuring that all my location entries have country specified, and are reasonably consistent to a next granularity within the country too. That took me many hours, over quite a few days, but I do now have a lot more confidence in my location details now.

I have only put in manual lat-lon details for a handful of locations, mainly where the historic address/region doesn't exist any more but I didn't want to loose the historic location information.

With my current ged-mapping, I am trying a different route ... like you, mapping from the original address to a more geolocatable address, and from there to lat-lon and mapping.

I am now part way through an experiment that I hope might go something like this:

1. Parse GEDCOM; fix any file record issues (Family Tree Maker is apparently notorious for broken GEDCOM export, and certainly has some non-compliant record structure in my tree export)
2. Add to a "FuzzyAddressBook" ... doing a fuzzy address match to see if there is an equivalent address already known; keep a full list of raw addresses and common fuzzy match address
3. Do any further fixes to the now shorter list of fuzzy match addresses, including any genuinely unique addresses
4. Perhaps also look-up/extend for implied country & region; and then geolocate to determine lat-lon
5. Save the translation table from incoming raw address to shorter fuzzy match + genuinely unique, and cleaned-up, locatable addresses; save this alongside the incoming GEDCOM to enable the translation to be efficiently reproduced in future if needed
6. Save the short-list (fuzzy+unique) address to country/continent/lat-lon lookup table; this also saved alongside the incoming GEDCOM to enable the translation to be efficiently reproduced in future if needed
7. Iterate around the steps above, using manual filters and summary stats to spot any gaps in country/continent mapping, failures to geolocate to lat-lon, suspicious or inconsistent address entries and geolocation; manually apply/tweak the alternate address values, and potentially manually add some lat-lon values where needed
8. Potentially also enable a "mega merge" of multiple address translation and geolocation tables to a central "know it all" geo-cache; but anticipate mainly using separate translation tables per GEDCOM/family context

I'm at the point now of needing to spend a bit of time trying out the fuzzy address mapping, and thinking about using implied country mapping and whether it is possible to automate consistency checking and proposing/adding missing higher levels of address hierarchy.

I have a concept in mind that addresses really ought to be read from right-to-left, not left-to-right ... so start at the country end, and then work down towards smaller locality, and only consider the finer details if it is proven to be consistent with the higher levels. But I'm not sure that it is practical to take the geolocation time overhead of doing it in exactly that mechanistic way for all entries, so I'm wondering about something like trying both ends ... full address, vs right-hand end (e.g. country), and only break it down further if the both-ends test yields a negative (inconsistent) result.

[colin0brass]

I've had a go at a script flow that takes 3 input files:

• gedcom file
• alternate address file (optional) ... specific per input gedcom file
• geolocation cache file (optional) ... specific per input gedcom file

In addition to a global cumulative geolocation cache file.

It seems to be working, though re-does more work than it should, when adding people having already parsed all the locations from the GEDCOM.

I also started looking at trying to extract a canonical address (including structured hierarchy for country, city etc), but haven't got very far with that yet.

I am also trying using 'libpostal' to resolve subtle variations in address syntax, but I interpret it doesn't work that well on partial and historic address values.

https://github.com/colin0brass/gedmap

To be continued ...

[D-Jeffrey]

I figured out how to refactor some of the ideas, (missing your well-structured code). I plan to add in the midlife points soon, reworking the the lifelines and adding parent/children lines. I was trying to use KML Schema to add structure KML data... not sure if that will cause issues outside of Google Earth Pro. Very limited testing on v0.2.6.8.

When I try and use some of my GED (from myheritage) with your code, it does not work. I get

2025-09-11 21:25:28,433 ERROR __main__: An error occurred: 'charmap' codec can't decode byte 0x9d in position 5526: character maps to <undefined>
Traceback (most recent call last):
  File "C:\Users\darre\source\repos\gedmap\gedmap.py", line 157, in <module>
    main()
  File "C:\Users\darre\source\repos\gedmap\gedmap.py", line 113, in main
    my_gedcom = GeolocatedGedcom(
                ^^^^^^^^^^^^^^^^^
  File "C:\Users\darre\source\repos\gedmap\gedcom.py", line 489, in __init__
    super().__init__(gedcom_file)
  File "C:\Users\darre\source\repos\gedmap\gedcom.py", line 425, in __init__
    self.gedcom_parser = GedcomParser(
                         ^^^^^^^^^^^^^
  File "C:\Users\darre\source\repos\gedmap\gedcom.py", line 196, in __init__
    self.gedcom_file = self.check_fix_gedcom(gedcom_file)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\darre\source\repos\gedmap\gedcom.py", line 206, in check_fix_gedcom
    changed = self.fix_gedcom_conc_cont_levels(input_path, temp_path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\darre\source\repos\gedmap\gedcom.py", line 224, in fix_gedcom_conc_cont_levels
    for raw in infile:
  File "C:\Users\darre\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5526: character maps to <undefined>

The file is ASCII and 0x9d and was associated with some cut and paste in the CONC related to a BIO.

[colin0brass]

Interesting. It sounds as though your GEDCOM file has some control characters in it, and my fix_gedcom_conc_cont_levels function is trying to open it as a plain text file and failing on that control character.

It looks as though we should be able to easily ignore illegal characters by adding "errors=replace" into the file open function call, e.g. changing from:

        try:
            with open(input_path, 'r', encoding='utf-8', newline='') as infile, \
                open(temp_path, 'w', encoding='utf-8', newline='') as outfile:

to something like:

        try:
            with open(input_path, 'r', encoding='utf-8', newline='', errors="replace") as infile, \
                open(temp_path, 'w', encoding='utf-8', newline='') as outfile:

If this doesn't work then we can try being more selective about the file format, or looking closer at the control characters in the file and deciding if we need to actually do something with them.

[D-Jeffrey]

That worked. as a fix for reading. It appeared to do a much better job of address resolution than my approach.

Your algorithm for finding the lat/long works much better than mine. It resolved 96% of my addresses. My approach was only 64%. A major improvement. I need to fold that into my geocoding.

Then I got this at the end.

2025-09-13 00:15:09,943 ERROR __main__: An error occurred: 'charmap' codec can't encode characters in position 16-19: character maps to <undefined>
Traceback (most recent call last):
  File "C:\Users\darre\source\repos\gedmap\gedmap.py", line 157, in <module>
    main()
  File "C:\Users\darre\source\repos\gedmap\gedmap.py", line 141, in main
    write_people_summary(args, my_gedcom.people, str(people_summary_file))
  File "C:\Users\darre\source\repos\gedmap\summary.py", line 90, in write_people_summary
    csv_writer.writerow([summary['ID'],
  File "C:\Users\darre\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 16-19: character maps to <undefined>

I hope it saved the cache along because that took 3 hours to work thru my addresses. ... oh good it did.

At the end I had KML, People and Places before this error.

I did a sort on the address list and had 7 duplicates. out of 1613 ... weird

Powell River,   British Columbia, Canada
Mountain View Cemetery, Lethbridge,   Alberta, Canada
Halifax Co Virginia
Fulton Co IL
Chapel En Le Frith, Derbyshire, England
Chapel En Le Frith, Chapel En Le Frith,   Derbyshire, England
Chapel en le Frith, Derbyshire

For Found_Country False here are some of the places. It still found Lat & Long but Found Country = False

University of Alberta
NW Sec 34 Twp 46 Rng 14 W4 (53.013980     -111.956964)
Vegreville, Alberta
Carragana, Saskatchewan
Magrath, Alberta
Angleterre
Mountain View Cemetery, Lethbridge. AB
Morristown, New York
Chilliwack, BC
Toronto
Botha Alberta Canada
Taber,Alberta,Alberta
Taber, Alberta
Lethbridge, Alberta, Alberta
Saskatoon, Saskatchewan, Saskatchewan
Powell River
Saratoga Co. New York
Auckland University
Middlemore Hospital, Auckland
Londres, United Kingdom of Great Britain and Northern Ireland
Ambler, Montgomery, Pennsylvania, United States of America
Ontario
Wortley, Leeds, Yorkshire, United Kingdon
Northumberland, Ontario
Muskoka Canada
Wootton
Can
North Augusta, Grenville, Ontario
New York
Brandon, Brandon, Mb
Chilliwack, British Columbia
Vancouver, BC
Quesnel, BC
Quesnel, British Columbia
Vancouver, British Columbia, British Columbia
Lacombe, Alberta
Township 47 range 14 Section 10 SW, Median 4 SW of Viking   53.034864     -111.972505
Township 51 range 8 Section 30 NE W5 ( 53.435995     -115.162270)
Bugsworth, Derbyshire
chapel en le frith derbyshire
Chapel en le Frith, Derbyshire
Hanna, Alberta
North Battleford, Saskatcehwan
Edmonton, Alberta
Bugbrooke,  Northamptonshire
Upper Canada
Of, Ottawa Valley, Ontario
Franklin Twp,Armstrong,Pennsylvania
Écosse
Eau Claire, Eau Claire, Wisconsin
Colfax, Dunn, Wisconsin
Brackley, Sulgrave
Loddon, Norfolk
Ocala, florida
Sterling, Alberta
Coeur d'Alene, Idaho
Pincher Creek, Alberta
East Meets West Chinley
Derbyshire
Chinley Bugsworth Derbyshire
Haut-Canada
Louisville, NY
United States of  America
Freeport, Stephenson, Illinois, United States of America
Ross County, Ohio
Alberta
Rock Co. WI
Långaryd i Jönk. l.

[D-Jeffrey]

That make a HUGE difference in the KML output. That really changes the richness of the output

compared to mine

[colin0brass]

Oh good, I'm glad it is (nearly) working for you, and giving useful output and ideas.

I suspect that means your GEDCOM data includes non-ASCII characters (e.g. unicode) in the people information, and so has exposed a fragility that I hadn't spotted with my own data.

From the error on line 90 of file summary.py, in function write_people_summary(), it looks as though this simple update should be able to enable handling of unicode characters. Add encoding='utf-8' into the file open command, e.g. change (lines 89 & 90 of the above file) from:

        with open(output_file, 'w', newline='') as csvfile:
            csv_writer = csv.writer(csvfile, dialect='excel')

to:

        with open(output_file, 'w', newline='', encoding='utf-8') as csvfile:
            csv_writer = csv.writer(csvfile, dialect='excel')

I have gone through my code and added a few more cases of utf-8 support just in case and pushed them to my GitHub repo.

On country detection, I currently rely on a country name lookup in all known country names from https://pypi.org/project/pycountry and any extras that I added or renamed through my geo_config.py and geo_config.yaml, or more specific alternate address insertion through <input filename>_alt.csv.

It is on my mind to add a way to infer country in a more flexible way, but I haven't ventured to do anything with this yet. Some loose thoughts and ideas as follows:

1. Explicit country lookup from address (after string clean-up, consistency improvement, disambiguation, alternate/improved address insertion etc)
2. Extend/infer address to add country & continent for recognisable location, e.g. from manual/scripted input (place dictionary substitution), or online geocoding service
3. Lat/Lon lookup to country & continent; I had a very lightweight brush/attempt with this via an AI service and it kept on insisting places in north Norway were in Russia, no matter how I tried to tell it otherwise; I suspect we would benefit from something like this, but I haven't got very far yet

[D-Jeffrey]

I reinstall the GIT and now it need postal. Seems like I'm missing a few setup parts for postal. Postal is harder to install on Windows than Mac.
I assume I need to install this first https://github.com/openvenues/libpostal. I'm really not a fan of MSys2/MinGW shim and all that is required to recompile it to make it work.

I assume libpostal will enrich the results, but I'm not ready to go down that path yet.

I rolled back to https://github.com/colin0brass/gedmap/tree/86559c0f8c5a9ef6ef82f3c6d6368685b4fcd3fe (pre-postal) and applied the utl-8 fixes. I finally got the heatmap.

The Goe_cache is 97.8%. I'm preplex, you have the geocode lat/long but that does not come with the country associated with it?

On a different note, I logged back in to myHeritage and the list of analysis they do has improve.

Here is the list #9

[colin0brass]

Thanks; I've tried switching from libpostal to deepparse. I haven't paused to examine how good it is, but from a quick test it seems to be working. See latest on GitHub.

[D-Jeffrey]

I have start to refactor and add your code in pretty much unmodified. I refactored the orginal code to include you LatLon replacing Pos.

So far I have location (split into location and latlon), geocode, goecache. I will work to connecting into your files as is, as much as possible... via a rework of geolocate_all and associates parse people.

I tried to look for an alternate, glad you found one that fit in so well. That is big set of packages it pulled in. Let see what this does (it did the requirements without issues).

It resolved the addresses so fast, 97.8% of the addresses found a geocode address (not sure if they are all right as there are 1600+)

So it ended up withe same heat map as before, but it completed in just under 9 minutes??? I had cleared out all the file and run it again... that was Superturbo charged. Pretty amazing! In fact I wonder if doing any lookup on OpenStreet map is even required, since they all seem to come back without any better result.

I'm not sure the found_country is that or found location. For example...

count	latitude	longitude	found_country	place	country_name	continent
1	52.44285	-109.155	TRUE	Unity, Saskatchewan, Canada	Canada	North America
1	53.01313	-112.841	TRUE	Camrose, Alberta
1	53.5	-112.05	TRUE	Vegreville, Alberta
1	52.58052	-103.105	TRUE	Carragana, Saskatchewan

All those addresses are correct, but the country should be Canada for all of them. But the lat/long is what it should be.

I was expecting the model memory size to be much larger but it was only about 600 MB.
There is a deprecated error at the startup of pymagnitudelight

Wow... just wow.

[colin0brass]

Sounds good. I will take a closer look at the caching and country look-up.

For the surprisingly fast speed, had you definitely spotted and deleted/renamed both the global and specific geo-cache files? Both in the input folder.

I did agonise over whether to have a global geo-cache and specific one per input file ... I instinctively worry about pollution, complication and risk of unpredictable and difficult to reproduce geocoding over time depending on which other GEDCOM files have been processed and updated the geo-cache.

File paths:

        logger.info(f'Processing GEDCOM file: {gedcom_file}')
        global_geo_cache_path = input_path.parent / args.geo_cache_filename
        alt_place_file_path = input_path.parent / f"{base_file_name}{FILE_ALT_PLACE_FILENAME_SUFFIX}"
        file_geo_cache_path = input_path.parent / f"{base_file_name}{FILE_GEOCACHE_FILENAME_SUFFIX}"

[D-Jeffrey]

oh... and here I thought your AI improvedment had been the magic. Ya, it is not so quick I was thinking that you LLM blob also had a lot of world addresses compressed into it. It seemed pretty small to be able to do that. It is back to the slow lookup after I dropped the global Geo cache file. I was thinking it was in the same .cache directory where the LLM gets loaded down into on the first run. I think the global cache is needed. Once you have that address, it is not like OpenStreet is going to change it's answer.. the only shift is when your alternate looks produces an different order of lookups that you might need to purge global and start over.

[D-Jeffrey]

Okay... was longer 5 hour 30 minutes (not so turbo). Seems to produce the same results without a 'Unknown' section (just skipping them in the heatmap)

So the resolved (for me) was very slightly better before the LLM. But have not

Resolved 96.9%

This is an example of what the old function (fuyzzy) could resolve (and the new LLM not):

6,52.4715652,1.0779969,True,"Market Green, Depwade Union Workhouse, Pulham St Mary Magdalene in The County of Norfolk, 
2,42.8620571,-70.9455769,True,"Williamsburg, Williamsburgh, Dundas, Upper Canada, British Colonial America",,Unknown
1,51.5074456,-0.1277653,True,"Londres, United Kingdom of Great Britain and Northern Ireland",,Unknown
1,42.8620571,-70.9455769,True,"North Augusta, Augusta, Grenville, Canada West, British Colonial America",,Unknown
3,42.8620571,-70.9455769,True,"Williamstown, Charlottenburg, Glengarry, Upper Canada, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"Jasper, Merrickville-Wolford, Leeds and Grenville, Canada West, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"Brockville, Elizabethtown, Leeds, Canada West, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"First Congregational Church, New Haven, New Haven, Connecticut, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"Center Church on the Green Church, New Haven, New Haven, Connecticut, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"First Congregational Church, Meriden, New Haven, Connecticut Colony, British Colonial America",,Unknown
1,37.379388,-122.1152034,True,"First Congregational Church, New Haven, New Haven County, Connecticut Colony, (Present USA)",,Unknown
1,42.8620571,-70.9455769,True,"Grove Street Cemetery, New Haven, New Haven, Connecticut Colony, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"Center Church on the Green Cemetery, New Haven, New Haven County, Colony of Connecticut, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"Plymouth Burying Ground, Plymouth, Litchfield, Connecticut, British Colonial America",,Unknown
1,42.8620571,-70.9455769,True,"Center Church On The Green, New Haven, Connecticut, British Colonial America",,Unknown

[D-Jeffrey]

I'm still trying to fold your code into this project. I have to say, you are programming at least two level of skill above my poor procedural python codings style. I have done C++ and C# but never took formal training in Python and it shows. Everytime I see a super(), I'm overwhelmed, but it. I'm tried to split geocode.py and splice it into replace my gpslookup. Each time I try and debug thru it, the class inhertance is not correct, or the split use of your GeolocatedGedcom class, need to be referernece to my GedcomParser... as I tried to keep my Gedcom parsing... as well, I'm trying to move to a different release (pre LLM and pre postal). As I do the merge, I'm trying to preserve you code based and orginal files as much as possible. There are few items which are cause modes to the code, Pos -> LatLon, Human -> Person...
it makes my head explode each time I try and do the splice. Your code is so much better, I will try and back off and try again, this time upgrade your GeolocatedGedcom to replace my GeocomParser....

[colin0brass]

Best wishes, I'm sure you will get there. (I'm just off to Austin, Tx for the week)

I'm on my own journey with Python too ... I have always been HW/SW adjacent ... more on the HW and SoC architecture side, but enjoying the overlap with SW. I have brushed with various programming languages over the years, and had ideas and some attempts/experience about what might make well structured code, but precious little time to practice it.

I deliberately used a few new techniques here which I could imagine might cause headaches for others, for example __slots__ in class declarations. I have never used this before, but found that I was being inconsistent in use of class variables, and accidentally creating and using new class globals after instantiation. I wanted to find a way to enforce consistency, without encumbering everything with having to go via class methods (function calls). I liked what I felt was the simplicity and elegance of directly accessing class variables after instantiation, and didn't want to loose that. I don't know if there are other ways to solve it, but I found out about __slots__ and chose to use it.

With super(), which I have used in at least here: class GeolocatedGedcom(Gedcom): you might well have figured it out already, but my simple thoughts and attempted explanation is something like this:

I wanted to add geolocation capability on top of standard gedcom parsing. I could have added it in the same Gedcom class, but that felt like over-complicating what could stay more simple. So I chose to use inheritance and overload the gedcom class to add a new GeoLocatedGedcom class, based on the simple original Gedcom class.

So now inside the new GeoLocatedGedcom class, we can (and do) refer up to methods (functions) in the parent Gedcom class using super().<reference inside parent class>

I have tried to use classes to separate different aspects of the functionality, and to partition the files as best I could ... though it did evolve a bit, and I struggled with how much to stick with location attributes which happen to be discovered via the GEDCOM files, vs trying to map them into my own data structures. What to do when I wasn't so keen with the GEDCOM line-up of attributes? ... live with it, or re-map? ...

I used to do all my coding in a plain text editor, and old-school print debugging on the command line, but have recently started using Microsoft Visual Studio Code which can be a great help with debugging (e.g. break-points, stepping and variable inspection).

[D-Jeffrey]

No way I would have made it this for with VS Code. I really like the slots rules enforcement in the Class. A wonderful and very useful lesson for me, which will cause me to go back and refactor more code. I have figure out super(), still makes my brain shock to process thru it.

I have shifted the code base to Person and LatLon so I can better fold in your gedcom.py. I'm a fan of re-mapping the Gedcom to enrich the relationship and summarization options.

Okay diving in to try and merge again.

[D-Jeffrey]

Well I finally managed to merge it in. I have a few more class re-factors to clear and then I will push v2.7 this weekend. I managed to break a bunch of things that I will need to put back together.

[D-Jeffrey]

Well I slammed it all together... not sure it was a great idea to do a branch for that, but I should be able to push that into the main after some more testing.