* FH-RM-Ancestry Sync to exploit hints (update)

Importing from or exporting to another genealogy program. This is the place to ask.
Post Reply
avatar
Mark1834
Megastar
Posts: 970
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 » 06 Oct 2021 11:09

This is a direct continuation of the earlier discussion of syncing FH with Ancestry via RM to exploit hints, but I've started a new thread to avoid having to scroll through three pages of now largely irrelevant material. It does, however, build on the approach suggested by Shosh Kalson in the previous thread.

The existing process is not very elegant, and requires many more manual steps than is desirable. Now that FH7 has added support for the UniqueID tag and the ability to connect to RM databases directly, there is an opportunity to improve this considerably.

I have a draft plugin that I am testing, but before posting it here, it is worth outlining the new workflow to get feedback from potential users. The existing plugin has had nearly a hundred downloads, but there has been relatively little user feedback. I don't know if that is because it is working perfectly, or users have given up on it as too complex.

There are two steps to the process - updating RM to reflect changes in FH, and syncing those changes with Ancestry. The first step was the weaker one, due to limitations in the RM file comparison tool, but that has now been largely automated by reading RM data directly from FH. The subsequent sync with Ancestry is still manual, but we have no access to the Ancestry API to modify that. However, it is now much better guided.

These screen grabs illustrate how it is intended to work, where the core feature of the new plugin is an audit report of all the differences between FH and RM, saved as a Research Note.

1. Export a GEDCOM from FH into RM using the plugin one-click approach to generating a sanitised GEDCOM extract. It doesn’t matter whether you use RM7 or the new RM8 - it works identically in both versions. The plugin demonstrates that the two files are equivalent, using the Sample Project as an example.
Original.PNG
Original.PNG (26.57 KiB) Viewed 622 times
2. Continue with your research until you are ready to update Ancestry. I have married off Judy Susan Munro, adding a husband and three children, updated a range of facts for other individuals, and detached Amanda Morag Munro from her parents, as further research has indicated that she is not their daughter.

3. Run the plugin to generate an accurate and complete list of all the differences between FH and RM.
Pre-update.PNG
Pre-update.PNG (39.77 KiB) Viewed 622 times
4. Generate a new export GEDCOM (one-click via the plugin), and import into the existing RM file. Use Share Merge in RM (which is designed for exactly this scenario, updating the database from an external GEDCOM) to automatically add new individuals and data and create family links. The existing process using RM File Compare requires all new family links to be created manually, which is tedious and error-prone, as well as generating many false positives in its assessment of which records have changed.

5. Run the plugin a second time to list any remaining differences that need to be tidied up manually, as FH is limited in its ability to create, link and delete RM records.
Post-update.PNG
Post-update.PNG (28.99 KiB) Viewed 622 times
6. Finally, update Ancestry from RM, using the audit report generated in step 3 as a complete and accurate guide of which records need syncing.

I want to give the plugin a good test before posting it using real rather than just test data, but comments from potential users are invited. Any interest?

To keep it focused, I suggest we don't revisit the “hints are useful/waste of time, do it this way instead” discussion. We've had that already (several times... ;)).
Mark Draper

avatar
David Potter
Megastar
Posts: 872
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by David Potter » 06 Oct 2021 13:26

Hi Mark

This sounds very promising - I need to read the post a few times over before any comments. But certainly I am an interested party.
I made some progress with the tools you recommended yesterday: firstly creating the UID's in FH7 and then running your formatting Plug-In choosing option 3. Then using the FH7 Gedcom Export function to create a GEDCOM with the Default option selected for the target App. Imported this GEDCOM into RM8 and managed to confirm using SQLite that the UID matched across FH7 and RM8. I then ran the RM8 import once again and imported the exact same GEDCOM to see if RM8 through up any messages regards the duplicate UID's but it did not. Lastly I ran the Auto Merge to see if the UID might now play a part in that process and as you have pointed out before it does not. So I ended up with a few hundred additions where the Auto Merge probably couldn't distinguish enough to suggest a Merge. IE, insufficient facts to work with for these Individuals, I'm not sure if that is the true reason though as to why it did not Merge. (I'm still going through a major Do-Over now FH7 exists) so some individuals are without Facts.

Next steps
I'll read up on that Share Merge feature to get familiar with it and test it out.
I was considering approaching RM8 development to ask if the UID could become part of the Merge processing as an optional choice - what do you think?

Cheers - David

User avatar
fhtess65
Famous
Posts: 134
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by fhtess65 » 06 Oct 2021 15:30

Just so I'm clear, all this happens within your plug-in? If so, that's very cool. Will be busy for the next couple of weeks but would love to play with this plugin as overall I find FH to be the superior program, but also want to keep my Ancestry tree current without retyping.

Also, is there a way in the plugin to move info from RM to FH? Or is it just one way, FH to RM? I admit to having a fear of keeping all my eggs in one basket.

Thanks,

Teresa
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com

User avatar
fhtess65
Famous
Posts: 134
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by fhtess65 » 06 Oct 2021 15:56

Further question - I assume it's safer to use the plugin in a copy of my project, not the master one?

Teresa
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com

avatar
Mark1834
Megastar
Posts: 970
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 » 06 Oct 2021 16:31

All in the plugin, Teresa :). But to be clear on scope - it is not designed to keep a full project replicated across FH, RM, and Ancestry, but to manage an extract (key standard facts only for non-living/private individuals, with no media, sources, or other frills), such as you would use for hint generation, cousin-bait, or DNA matching. The source data always comes from FH.

Added in edit - if all you want is a copy of your tree on Ancestry, any GEDCOM export will do, either the full fat Export Gedcom plugin (most options), standard FH export, or my plugin (simple sanitisation). It’s only worth going via RM if you want to preserve hint history.
Last edited by Mark1834 on 06 Oct 2021 16:39, edited 1 time in total.
Mark Draper

User avatar
ColeValleyGirl
Megastar
Posts: 3148
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by ColeValleyGirl » 06 Oct 2021 16:34

I'm looking forward to using this if/when I ever get round to updating my research.

User avatar
tatewise
Megastar
Posts: 22518
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise » 06 Oct 2021 17:02

Mark1834 wrote:
06 Oct 2021 16:31
It’s only worth going via RM if you want to preserve hint history.
You would need to go to RM if Media are important because Ancestry does not bulk upload Media.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
fhtess65
Famous
Posts: 134
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by fhtess65 » 06 Oct 2021 20:36

All understood - I will play with it after my vacation (will be visiting someone, so little time for genealogy) and see how it goes.

Thanks for the information :)

Teresa

Mark1834 wrote:
06 Oct 2021 16:31
All in the plugin, Teresa :). But to be clear on scope - it is not designed to keep a full project replicated across FH, RM, and Ancestry, but to manage an extract (key standard facts only for non-living/private individuals, with no media, sources, or other frills), such as you would use for hint generation, cousin-bait, or DNA matching. The source data always comes from FH.

Added in edit - if all you want is a copy of your tree on Ancestry, any GEDCOM export will do, either the full fat Export Gedcom plugin (most options), standard FH export, or my plugin (simple sanitisation). It’s only worth going via RM if you want to preserve hint history.
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com

avatar
David Potter
Megastar
Posts: 872
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by David Potter » 10 Oct 2021 14:37

Hi Mark
I have read this over and don't have any further comments apart from when can we test the new Plug-In?

Thank you.

avatar
Mark1834
Megastar
Posts: 970
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 » 10 Oct 2021 19:19

Thanks David. I’ve been exploring potential additional options, such as allowing living people to be copied, synthesising missing birth and death data, etc, but decided in the end that they add too much complexity for a first prototype. The whole point of this plugin is that it produces a consistent output without having to remember multiple settings.

The list of events to be included will be easily modified (and can include custom events) by a simple edit of the plugin script.

As it stands, the plugin isn’t a drop-in replacement of the earlier KB version for existing users, as the existing RM UniqueID values will need to be used to update the FH project to maintain the link between apps. For my main project, that wasn’t an issue, as it uses a unique Custom ID for each individual that I made sure I exported right from the beginning for just this eventuality. It was a simple additional script to match individuals on Custom ID and update the UniqueID accordingly. I have another mid-size project (~ 1000 people) that doesn’t use Custom IDs, so I will need a more generic solution for that.

My idea at the moment is to match individuals on full name, lifespan dates and spouse full name. That should take care of most people.

I’ve got a busy week coming up, so it might be next weekend before it’s ready for posting.
Mark Draper

avatar
David Potter
Megastar
Posts: 872
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by David Potter » 11 Oct 2021 09:17

Thank you Mark. That sounds good and gives me time to finish my last few Re-Key's in my tree under FH7.

Best wishes

David

avatar
Mark1834
Megastar
Posts: 970
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 » 17 Oct 2021 20:03

Prototype plugin attached. Use as per earlier postings. Additional points to note:
  • The plugin takes care of formatting UniqueID values, so there is no need to convert to an RM-compatible format prior to export.
  • It works equally in RM7 or RM8, as the relevant tables are unchanged.
  • It does not write anything to your project apart from the Research Note audit report, but as ever, it will be safer to use on a copy, not the original. Also best not to risk your live Ancestry sync if you already use the original version, so generate a new RM database for testing.
  • The help button brings up the relevant page on the KB, but this has not yet been fully updated for the new plugin.
  • The list of Facts exported and compared can be changed by editing the plugin script at lines 112 to 119. Instructions are included in the plugin.
  • There is no enhanced memory management included in the plugin (which is often needed in FH7 for large datasets), but it is not anticipated that very large projects will be synced to Ancestry (if you do, you’ll be spending a LOT of time reviewing hints!).
Remember, this is NOT a general FH-RM comparison tool, and should be used only in the context of the plugin. Over to potential users to take for a test drive…
Attachments
Family Historian - RootsMagic - Ancestry Sync (0.1).fh_lua
(39.97 KiB) Downloaded 5 times
Mark Draper

Post Reply