* FH-RM-Ancestry Sync to exploit hints (update)

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
avatar
StockportGerbil
Gold
Posts: 12
Joined: 29 Nov 2017 18:08
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by StockportGerbil »

Firstly - thanks very much for your patience - it is appreciated.

I spent the morning trying again and again made a mess. I'll try again this evening but I think I know what I did wrong and it is me being gormless! I'll report back :-)
avatar
StockportGerbil
Gold
Posts: 12
Joined: 29 Nov 2017 18:08
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by StockportGerbil »

It was sort of me being gormless (getting into a mess with file locations) and it's now working.

For reasons I don't understand when I exported the initial gedcom from Family Historian to Rootsmagic and imported it, my father's parents weren't shown in rootsmagic. I updated the script to allow living and private individuals to be exported and then it worked. (I had ensured there wasn't a private flag set on any of the three people involved and whilst I am happy to say my dad is still living I haven't explicitly set the living flag.)

Anyway, thanks again for your help and patience
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

RM and FH behave differently on general file import and export. RM always opens file selection at the defined default folders, whereas FH generally has a “last used” convention, which I follow in the plugin. Neither is intrinsically better than the other, they are just different.

“Living” flags can be slightly tricky, as RM neither imports nor exports them in GEDCOM and makes up its own mind whether they should be set. The next plugin version will proactively force all RM flags to the FH values.
Mark Draper
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Version 0.3 of the plugin is attached, and is a recommended upgrade for all users. There are a number of changes from the last public version, many as a result of gaining experience with how RM operates.
  • Full automatic support for users’ preferred font and zoom level. It has no effect on how the plugin functions, but makes the plugin window look and behave just like any other FH window. It’s an easy change to make, so I will incorporate that into all my future plugin revisions.
  • Full support for all languages in both the FH project and RM database, not just the Western Europe workaround described earlier in this thread.
  • RM has no “undo” or snapshot capability, so I have added an option to create an automatic backup copy of the RM file before any changes are made (default = yes).
  • Option to display report output as either plain text or hyperlinks (default = plain text). Links may be more convenient, but cause the “Updated” stamp to be modified, which some users don’t want. This is due to be changed in FH 7.0.10, but I’ve left the option in for users who may prefer plain text anyway.
  • Option to include living people without hacking the code (default = no).
  • RM does not update record timestamps if they are changed by merging, so the plugin forces an update of all changed records. This has no effect on the subsequent Ancestry sync, but keeps the displayed edit dates accurate.
  • RM is less happy with missing individuals than FH. If somebody has multiple children by (presumed) different unknown partners, the families can get amalgamated when merging. The plugin now automatically deletes any incorrect links created during the merge.
  • RM does not import or export the Living flag, and makes its own decision on whether this should be set or not. The plugin now forces all RM values to match FH during the Update step.
  • The Compare step is now read-only, and makes no changes to the RM database. Previously, it did some silent housekeeping.
  • RM merge can create duplicate place records. Unfortunately, the plugin cannot delete these, but it lists them for optional manual deletion. Leaving them in makes no difference to the database, they are just like FH unused place records.
  • Individual Facts marked with the Private flag are now excluded from the GEDCOM export. Any such facts inadvertently exported previously will be deleted from RM during the Update step.
There appear to be about 30 users who downloaded the first prototype plugin, but not the subsequent update. I'd be interested in any feedback as to why you did not continue with it. It's now much more intuitively structured than the first version, so have another go if you feel it's still something you want to do but were put off by potential complexity.
Mark Draper
avatar
HanniganResearch
Gold
Posts: 19
Joined: 10 Jan 2021 18:53
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by HanniganResearch »

Thank you for the work on the plug-in to resolve syncing an Ancestry tree with data in FH7. I was happy to find out that RM8 data is natively read into FH7, although with some limitation.

I have read through the 54 posts in this thread and follow the logic involved. However, does the UniqueID plug-in need to be run initially when processing in FH7?

My process is to take the Ancestry tree (primarily focused on DNA matches/connection) and sync with RM8 - then import the RM8 datafile into FH7 for further work. At this point (I think) the UniqueID process is added before beginning the export process back to Ancestry tree, is this correct?

I have not attempted to use the plug-in yet, just read through the data to get an understanding of the process before attempting it.

Thank you for your patience and understanding.
User avatar
ADC65
Superstar
Posts: 457
Joined: 09 Jul 2007 10:27
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by ADC65 »

I have been using your v0.2 plugin for a couple of weeks and would like to say thanks. I will update to v0.3 and let you know if I have any problems, although the listed items are not ones that affected me particularly.

Once you get to grips with how the Plugin works, and the workflow required for the updates, it is quite easy. However I think the actual instructions on the KB and within the Plugin could do with a bit of tweaking to make it easier to understand for users that have not read this thread in entirety. I will try and make some suggestions in the near future.

Nothing to do with the Plugin, but I was irritated to find how clunky the update is from RM8 to Ancestry, or maybe I'm missing something. It appears you have to check and confirm every single change for each person. Ideally I would like RM8 just to make all the RM8 > Ancestry changes without having to go through the whole rigmarole - especially as I usually have quite a few updates. I thought RootsMagic did this, but perhaps I am thinking of Family Tree Maker?

Anyway, thanks for the work you have done on this.
Adrian Cook
Researching Cook, Summers, Phipps and Bradford, mainly in Wales and the South West of England
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

The plugin is mainly intended to take an existing FH tree, and export to RM and then on to Ancestry. However, what you are intending to do sounds possible, but might need a little more work.

Could you clarify exactly what you have now please, to advise the best way forward? It sounds like your "master copy" is in Ancestry, which is synced with RM8 via TreeShare, then exported to FH. Is that correct? If you make changes to your tree, do you do that in Ancestry, RM8, or FH? How do you keep FH and RM synchronised at the moment?

You don't mention that you have read the associated KB page. That gives a useful overview of when it is best to use this plugin (essentially to preserve your hint history), or where simpler solutions would be adequate.
Mark Draper
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

@Adrian - please do make suggestions, probably best in the Maintaining the KB topic. It is a complex topic, and I might be a bit close to it to see the issues a newcomer may struggle with. In general, the KB should be regarded as the "master" instruction page. Forum pages have a habit of wandering down rabbit holes and tend to focus on immediate issues rather than the overview... :)

Yes - FTM has full two-way Ancestry sync, but doesn't sit alongside FH very well. RM is a much better fit, but has more limited sync. I suspect that MacKiev may have been granted exclusive rights to the full sync when they purchased FTM from Ancestry.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

@HanniganResearch: I don't believe the Format UniqueID plugin is needed here as the RM UniqueID is imported to and exported from FH in RM compatible format. If am wrong, no doubt somebody will correct me.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ADC65
Superstar
Posts: 457
Joined: 09 Jul 2007 10:27
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by ADC65 »

Thanks for confirming that Mark. If you could just quickly write a plugin for FTM now then please :lol:

I will do as you suggest and post any suggestions to the KB thread. I hope I didn't come across as critical, I was impressed with the documentation and the explanations it contained.
Adrian Cook
Researching Cook, Summers, Phipps and Bradford, mainly in Wales and the South West of England
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Not at all. In principle, the plugin could interact with other apps, but MacKiev seem much less open about their data model than RM. I don’t know of any other app that does a native import of current FTM, and the underlying database may even be encrypted to prevent access by anything other than FTM. For me, that’s good enough reason not to use it...
Mark Draper
avatar
HanniganResearch
Gold
Posts: 19
Joined: 10 Jan 2021 18:53
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by HanniganResearch »

Thank you Mark for your help.

Currently, I use Treeshare to download a tree from Ancestry into RM8 then use the import into FH7. I work in FH7 because the interface is much superior to RM8. I make updates to the data in FH7 and then have to manually add the changes to Ancestry. After confirming some hints on Ancestry I begin the process again, however import a new RM8 to compare with the current master FH7 to keep it updated.

It would appear the plugin would be a much simpler process.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Thanks - that sounds like a fairly complex process, with updates being made in both FH and Ancestry. In general, it is better to always update just one version of your tree (usually FH), and copy that to other locations. You don't mention sources or media - could they get added in either location and copied across as well?

The first step is to recognise that the plugin is designed to manage a "bare bones" copy of your tree on Ancestry - just key facts, with no sources, notes or media. That is sufficient for hint generation, DNA matching, or general "cousin bait" visibility. If you want a "full fat" Ancestry tree, complete with media, sources, and full details of your research, this is not the route to take. Managing a full detailed tree such as that by plugin would be extremely complex, and carry a high risk of data corruption if done incorrectly.

Assuming your current FH tree is up to date and complete, by far the easiest option is to start using the plugin, and upload to a new Ancestry tree. You will lose your existing hint history, but only you can judge whether that is a showstopper or not. Reject all hints (even if they are correct), and make all your changes directly in FH. Use the plugin to feed updates to RM and hence to Ancestry.

You can try it out without commitment, just leave your old tree in place until you are confident that you want to continue that way of working.

Does that help you decide the best way forward?
Mark Draper
avatar
HanniganResearch
Gold
Posts: 19
Joined: 10 Jan 2021 18:53
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by HanniganResearch »

Thank you Mark for the detailed response. This make perfect sense. I like the interface of FH7 a lot better that RM8, but it is practically only useful for the TreeShare feature (IMHO).

Any suggestion on how to get the WebTags (i.e., DNA Match, DNA Connection, DNA Common Ancestor) from Ancestry because RM8 is incapable for grabbing them? After taking the time to tag the Ancestry file, it is important to gather that information, albeit you would probably suggest update the FH7 data - would that be a List to be created?

Thank you again for your help.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Have a look at this thread for some ideas on tree tags.

The Ancestry Tree Search gives you the ability to list individuals by tag, so you can generate crib-sheets of your existing tags. I can't think of any way of syncing them between FH and Ancestry automatically, so it may be best to record the information on just one side, either as a List/Flag in FH or re-establish as tags in your new Ancestry tree.
Mark Draper
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

i get the following error on the last step:

[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:930:
attempt to index a nil value (local 'Frm'). No changes have
been made to data records.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Sorry about that. It’s in the new check for RM individuals that have been incorrectly linked to multiple one-parent families during an RM merge. I can’t see immediately what the problem is, but I’ll have a more detailed look tomorrow.
Mark Draper
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

no problem. Rather, thank you for your commitment and creation of the great plugin
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

This plugin function searches the RM file for children linked to multiple families, each of which has a missing parent. It is designed to correct a flaw in the RM Merge, whereby if an individual has multiple children by unknown and presumed different partners (typically, but not always, an unmarried woman), all children become linked to each family.

In this case, the error arises because one of the linked families has both parents missing. The plugin constructs the RM family table by examining an individual's spouses. This is more complex than simply copying the FH family table, but it makes maintaining correct spouse order much easier. If a family has neither husband nor wife (in FH/GEDCOM terminology, father or mother in RM), no family record is created, even though it is still linked as the parent family of a child.

It might be a few days before I can explore that in detail, as it will need a fair bit of experimenting to see how RM handles different scenarios, but in the meantime the simple workaround is to modify the plugin script to omit that check. Comment out (with two dashes) the function call, so line 453 becomes as follows. The function arguments are the four FH/RM individual and family tables, the collated list of UniqueIDs, and a link to the RM database.

Code: Select all

--    if Update then CheckOneParentFamilies(database, tblFHI, tblRMI, tblUID, tblFHF, tblRMF) end
The plugin should then run to completion. It will probably show individual(s) with mis-matching parents, but this has no effect on the subsequent Ancestry sync. It's best to leave them as they are until I have a permanent fix. You could delete them in RM (not easy in the RM8 interface :(), but they will come back next time you run a new Merge.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

Mark, I'm fascinated by your statement: "no family record is created, even though it is still linked as the parent family of a child." In FH, such no parent Family records do definitely exist. They have no HUSB or WIFE tags but do have one or more CHIL tags and are listed on the Records Window on the Families tab as ...of (parents not known). I can post examples if you like. It is even possible to have a Family record with no HUSB, WIFE, or CHIL tags at all.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Yes, I know that. I thought it was clear from the context that the issue is not what FH can do, but how the plugin and RM handle such records.

RM does not have the same priority for strict GEDCOM compliance as FH, and does not permit such entities to be created in the UI, even though it might import them in the GEDCOM. We can debate the relative merits of the two approaches until the proverbial cows come home, but they are what they are, and I just have to handle the interface between them.
Mark Draper
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

The latest update has been user tested without raising issues, so is ready for general release. Thanks to cschroeder for kindly sharing a large database of mostly Central European names that enabled me to address a couple of other issues as well.
  1. RM only partly supports families with no spouses (it can import them but not create them) and Ancestry does not support them at all, so they are now excluded from the GEDCOM export automatically and any existing examples deleted from your RM database when running an Update. This avoids any complications from RM mishandling them on merging.
  2. The RM GEDCOM import can move name text after the highlighted surname to the suffix field, even if not stored as such in FH. The plugin now allows for this minor difference in formatting.
  3. Sex is a required field in RM but optional in FH, and the plugin was not correctly handling records with no defined FH Sex. Now fixed.
  4. The check for multiple one-parent families introduced to correct RM merge issues was not working correctly when the common parent was the father. Now fixed.
  5. The is a new option to generate a list of matching FH-RM record numbers when running an Update that can be used to identify the correct RM individual from multiple entries with the same name and no other distinguishing facts.
  6. When running either a Compare or Update, the selected RM file is compared with the one used previously, and the user alerted if the new file has a different name or older timestamp. This helps guard against selecting the wrong file if the user has multiple test versions.
Mark Draper
avatar
StockportGerbil
Gold
Posts: 12
Joined: 29 Nov 2017 18:08
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by StockportGerbil »

Thank you for your efforts - they are appreciated :-)
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

Hi Mark,

is it possible to add an option to overwrite fields like name or dates like birth or baptism?

Most of the changes I have to make manually affect these fields.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Do you mean manual changes that are listed after running an Update? Dates should be overwritten automatically, as the new date is added when merging records, and Update deletes the old one. Unfortunately, the plugin cannot change name fields in RM as RM uses a proprietary format (RMNOCASE) to support UTF-8 names that the fhSQL() library used by the plugin can read but not update.

There is no direct overwrite of Ancestry data from the plugin, as FH have not been given access to the Ancestry API.
Mark Draper
Locked