* FH-RM-Ancestry Sync to exploit hints (update)

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

This updated version of the plugin has a number of significant enhancements over the original, mostly in terms of the user interface and how reports are presented.
  • A new main menu gives much more guidance on how to use the plugin, rather than just simple button options.
  • There is a clearer distinction between the first read-only comparison of FH and RM, which is used to report all the differences between FH and RM (and hence the linked ANC tree), and the subsequent write access that removes old versions of updated facts.
  • The generated Research Notes are reformatted with clearer instructions for how they are used, rather than just listing differences without comment.
  • All updates involving Family Facts are reported under the individual spouses, as this is how they are processed.
  • All changes are now reported, correcting the minor omissions noted earlier.
  • There was an inconsistency for Individuals who had been merged. FH keeps both Unique ID values, consistent with the approach of GEDCOM 7, and both were exported by the plugin. However, RM7/8 support only a single Unique ID value, and when reading in a GEDCOM file only the LAST value is retained. The plugin uses the FIRST value for FH records, leading to a mismatch when comparing records. This has now been fixed to export only the FIRST value to keep it consistent with what RM expects to see.
  • “Living” flags are exported and synced, as per earlier discussion.
Instructions for modifying the list of facts exported and whether Living or Private Individuals are included are given at the start of the plugin script (select More… and Edit.. from the Plugins Window).

The KB page has now been updated to describe this version of the plugin and sync process.

Finally, thanks to David Potter for his invaluable suggestions and testing as the new version was developed.

[Edit - I've temporarily deleted the download file, as there may be an issue with the Living flag. It's late now, so I'll investigate further tomorrow and repost - sorry about that ]
Last edited by Mark1834 on 18 Nov 2021 23:28, edited 1 time in total.
Mark Draper
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

I've put this in a separate post to avoid confusion. If anybody is currently using the original sync process and plugin, this simple additional plugin updates the Unique ID values in your project from your RM database to enable transition to the new version. Use is explained in the KB page linked above.
Attachments
Update UniqueID From RootsMagic (0.1).fh_lua
(2.64 KiB) Downloaded 95 times
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

Mark, a little feedback on my first use of the Family Historian - RootsMagic - Ancestry Sync (0.2) plugin.

The Step 2. Plugin Export button defaults to saving the GEDCOM in C:\ProgramData\Calico Pie\Family Historian\Plugins\
IMO that seems an unusual folder to save GEDCOM files.

The plugin function ExportGEDCOM() uses the iup.filedlg{ } predefined dialogue.
That could use the 'directory' or 'file' attributes to default to say the Project Public folder path using:
fhGetContextInfo("CI_PROJECT_PUBLIC_FOLDER")
It could also provide a default GEDCOM filename.

The Step 1. Plugin Compare button and Step 4. Plugin Update button could similarly default to a better folder than C:\ProgramData\Calico Pie\Family Historian\Plugins\
In my experience the RootsMagic projects are in the ...\Documents\RootsMagic\ folder.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Mike, that’s not correct. The iup dialogue is unconstrained, so defaults to the last used folder. Deliberate decision not to force an initial default, as any of the default RM folder, the Project Public folder, or indeed any other choice are all equally valid, so I don’t want to impose any particular choice on the user.

It works quite well in practice. I tend to use the Desktop for quick test files, and a dedicated folder for my live data.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

Yes, OK, it is better once the user starts using it, but if another plugin uses iup.filedlg{ } they will compete with each other.
The IUP documentation says iup.filedlg{ } will eventually forget the last used folder.
Also, the folder for Export is likely to be different from Compare and Update.
The way I handle such issues is by offering an initial default but save the chosen folder as a 'sticky' setting that gets used as the default next time the plugin is used. Then a different default can be saved for each of Export, Compare and Update.

BTW if the chosen folder has any accented characters such as ë then the plugin says it has saved the file but iup.filedlg{} has not saved anything. Use the following command to allow IUP to handle such folder paths:

iup.SetGlobal("UTF8MODE_FILE","YES")
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Sorted - when RM imports a GEDCOM file, it makes up its own mind on whether to set the Living flag or not (it is not a standard GEDCOM item, and RM ignores it for both export and import). Sometimes it is set for individuals with no death date even if they are the parent of somebody who died centuries ago! It wasn't apparent before, as the first plugin version automatically cleared all Living flags, but that step has been omitted now that they are permitted in the RM database.

There is nothing we can do about it, but the workaround will be to run an "Update" from the plugin before running TreeShare for the first time. This forces all Living flags in RM to the same value as in FH and ensures that correct values are uploaded to Ancestry.

There is another minor tweak that I want to make before reposting, so I'll repost it when I've done that.
Mark Draper
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

The plugin is ready to go again. Apart from the "Living" modification noted above, I have also incorporated Mike's suggestion to permit extended characters in file and folder names, and moved Custom ID to the optional facts for export. It is not strictly a Fact in FH, but this makes the export optional rather than fixed.

There was a report of RM links being disrupted by renumbering the Individual Records in FH to test the robustness of the process. I have not been able to reproduce this so far, but it illustrates the warning given in FH about the unpredictable effects of renumbering. If in doubt, don't do it!

[Attachment deleted as it has been superseded]
Mark Draper
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

Hi Mark,

i have the problem that after an initial gedcom export..
the compare function throws an error:

[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:492:
attemt to perform arithmetic on a nil value. No changes
have been made to data records.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

Mark, this the same problem as elsewhere regarding file paths with accented characters and needs:
lfs.attributes(fhConvertUTF8toANSI(FileName), 'modification')
and that form of conversion may be needed elsewhere.

cschroeder, please confirm the file path to your RootsMagic project has an accented character.

Tip: I arrange for file paths to always contain an accented character such as ë for testing.
This is a problem that I predicted and reported long ago when FH started supporting UTF-8.
Different library functions require file paths to be in UTF-8 or ANSI. It is a nightmare!
There was advice about it in the old KB that seems to have vanished from the new KB.
I also reported it to CP in 2020 as 'Unicode in Folder/File Paths in Plugins' ticket #138290.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

Hi tatewise,

yes... the path contains german "Umlaute" like "Stammbäume"...
moving the rootsmagic files to another directory and everything was fine...
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by ColeValleyGirl »

tatewise wrote: 27 Nov 2021 21:16 This is a problem that I predicted and reported long ago when FH started supporting UTF-8.
Different library functions require file paths to be in UTF-8 or ANSI. It is a nightmare!
Did you expect them to rewrite the 3rd party libraries?
There was advice about it in the old KB that seems to have vanished from the new KB.
I didn't remove any advice about UTF-8 when I migrated the contents about plugins -- perhaps you're misremembering, but if you can tell me which topic uou think it was in, I can go back and check).
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Apologies - that's what happens when somebody lives all their life in a small offshore island without a full appreciation that most people don't have English as their first language :).

I've checked the plugin code, and there is only one line that needs changing. This calls a third-party library that is not compatible with UTF characters. I've updated my master copy, but it is not worth posting a new version at this stage for just that simple change. Users who want to use extended characters in file and folder names can edit version 0.2.1 of the plugin themselves (click on More>> in the Plugins Window to reveal the edit button).

Line 492 currently reads

Code: Select all

local FileAge = os.time() - lfs.attributes(FileName, 'modification')
Change that to the following, and save the modified script

Code: Select all

local FileAge = os.time() - lfs.attributes(fhConvertUTF8toANSI(FileName), 'modification')
Version 0.1 of the plugin does not generate the error, as the errant code was added as a check on when the RM database was last updated.
Mark Draper
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by ColeValleyGirl »

Probably not worth changing anything if it wad the only use you're going to make of the library, but getFolderContents in fhFileUtils returns among other things the attributes of each file in a folder, without using the problematic libraries.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

I have just experimented with tblFolder = fhFileUtils.getFolderContents(strPath,false) and it returns modified and created timestamps in the format: "02/02/2021 18:15:33"

That is very different from the integer seconds format used by os.time() and lfs.attributes(FileName, 'modification') so will need some conversion to make them comparable.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

The Plugin is really great and easy to use…

One thing… i don’t know where the failure is…
I merged some persons in FH and made an export to Rootsmagic. Everything was fine and
after an auto merge the count of persons was correct.

But after a Sync to Ancestry the merged ones are duplicated in Ancestry online. Is this a
general problem of Rootsmagic or with the unique IDs we used?
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

You have effectively deleted records from FH by merging (the Individuals that were merged FROM), so these will be identified by the plugin as records that you need to delete manually in RM prior to opening TreeShare. Once you have done that, TreeShare will show them as records that exist in Ancestry and not RM and give you the option of deleting them. Have you done all those steps?

I have discovered another potential issue while trying to duplicate your problem. Depending on when Ancestry was last updated, it is possible that new facts that are in the deleted record may not get flagged as needing uploading to Ancestry, as that Individual is marked for deletion. It would be safest to run the sync process prior to merging, as well as after. I will add that to the plugin help.
Mark Draper
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by cschroeder »

oh, I probably forgot to delete people this time, sorry ..

But I still have 2 ideas for the plug-in:
- an option for the automatic creation of the unique ID
- Automatic opening of the "Research Notes"

It would also be great if you could take over the "Family Search IDs" in FH from Rootsmagic.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

It is possible in principle for the plugin to calculate and add missing UniqueID values automatically. However, it means interacting with the hardware in more detail than is allowed by current plugin functions (get the MAC address of the network card as a seed to ensure uniqueness, and read the system time to the nearest 100 ns). FH does the calculation already, so I have asked CP for an "fhCreateUniqueID()" function to make it easily accessible to a plugin. It is on their consideration list, but realistically probably not very near the top as it is rather niche.

Agree opening the Research Notes automatically would be good, but plugins can only manipulate data, not control FH directly, so that is not possible.

There was some discussion of Family Search ID in this thread. Currently, FH ignores the FamilySearch ID when doing a direct import from RM, even though it has a currently unused field of the same name. Even if FH is eventually updated to include it, it won't help users who have already imported from RM. Perhaps we do need that simple plugin to extract the value from RM and update FH records, matching on UniqueID?

There are two modifications to the plugin that I have made to my master copy but not yet posted. Currently, it exports all defined facts for non-living/private individuals and families, but it would be better to omit any fact marked as "Private". The update fixes that. It also now uses the default font and Screen Font Scaling introduced in FH 7.0.9 to make the appearance of the plugin window consistent with other windows in FH. It's an easy change to make, so it will be better to make that my default behaviour now on all new or updated plugins.

I don't like posting too many updates on FHUG as I can't easily delete old ones. However, if anybody needs the update now, please let me know.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

It is NOT strictly true that "FH ignores the FamilySearch ID when doing a direct import from RM".
FH imports the RM/Legacy/Heredis _FSFTID tag containing the FamilySearch ID.
A plugin can copy that to the FH _FSID tag and was discussed in that FamilySearch Id (19875) thread.
So users who have already imported from RM can utilize the imported _FSFTID / _FSID FamilySearch ID.
Last edited by tatewise on 11 Dec 2021 16:00, edited 1 time in total.
Reason: Misleading advice removed
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Mike, you are getting confused between the old style GEDCOM export/import and direct import. It is the GEDCOM export/export that creates the _FSFTID tag that is saved as a UDF. Direct import ignores it completely, as I said above and confirmed by another user in the thread you quoted.

The existing Change any Fact Tag plugin can be used by somebody who imported prior to FH 7.0.8.2, but can't do anything for new imports. They would need a short dedicated plugin, which would also work for GEDCOM importers provided they had not deleted the _UID tag, which FH previously treated as a UDF.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by tatewise »

OK, sorry.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
StockportGerbil
Gold
Posts: 12
Joined: 29 Nov 2017 18:08
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by StockportGerbil »

Sorry - I'm sure I'm being gormless here, but I'm trying to run the update process for the first time and when I come to import the updated gedcom file into rootsmagic 8 I can't see / find the share merge option in Rootsmagic.

Where should I be looking or am I missing something?

Thanks

Kevin
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

Not gormless at all Kevin - overall, I think I quite like the new RM8 interface style, but it could do with a bit of tidying up as it can be hard to locate the feature you want.

With your master RM database open, click on File > Import Data > Import from RM 1-7, GEDCOM, or another Program > GEDCOM (which is the only option available). RM lists recent files, but for a first update, click on Browse for File.

This adds the GEDCOM to your file, automatically adding all new individuals, relationships, and facts. The next step is merging all the duplicate individuals. Click on the People tab, then the Tools icon in the top right corner (the spanner and screwdriver). Then proceed as per the description in the KB.
Capture.PNG
Capture.PNG (21.58 KiB) Viewed 4332 times
I recommend that you take a backup copy of your master RM file before doing any of this, as RM has no "undo" if you make a mistake. The next version of the plugin (which is being user-tested at the moment and should be released in the near future) creates backups automatically, as well as including a number of other enhancements.
Mark Draper
avatar
StockportGerbil
Gold
Posts: 12
Joined: 29 Nov 2017 18:08
Family Historian: V7

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by StockportGerbil »

Thanks - I was looking for it on the file screen.

I tried following the process and it seems to have failed / every record has changed apparently. I'll try again from scratch and see if I can figure out what I'm doing wrong. :-)

Thanks again for your help
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: FH-RM-Ancestry Sync to exploit hints (update)

Post by Mark1834 »

That sounds very much like a mismatched database (wrong GEDCOM export?, wrong RM file?).

Starting with a copy of your FH project, work it through one step at a time:
  1. Export a GEDCOM file and import into RM as a new database. You must use this GEDCOM export, not the FH built-in or plugin versions, as they do not do the required extracting and sanitising.
  2. Run a Compare to check that the record sets match.
  3. Make a few simple changes to FH - add a new individual, update a couple of new facts for existing individuals, add a couple of new facts.
  4. Re-run Compare to confirm that the changes are recognised.
  5. Export a new GEDCOM, and import/merge in RM as described above.
  6. Run an Update to delete old facts from RM.
The Update cannot process all changes in RM, due to limitations in the code library supplied by CP. Deleting individuals, unlinking relationships, or updating name fields in RM has to be done within that app, but the plugin will identify what changes need making. The next update will do more processing, as I have explored the limits of what FH can do reliably.
Mark Draper
Locked