* Import of RM7 Citations

Writing and using plugins for Version 5 and above.
avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 18:33

Thanks, that makes sense. I’ll send you a PM. I’m getting increasingly confident that we are seeing anomalies in the RM export rather than plugin errors, so I’ll tweak the code to continue processing the file and list potential errors at the end, rather than aborting the process if it sees something it doesn’t understand.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 21:27

Files sent
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 22:34

Had a quick look tonight before packing up. My suspicions were correct. All the errors are data anomalies in the RM GEDCOM export, either a source level field being recorded at citation level, or a field being mis-named when cited.

The non-living extract you sent me generated three errors due to source/citation confusion, all for the same source record, and one mis-named field. It may not be coincidence that they all relate to the same individual, Amelia Auvil. They are different errors to the ones you reported on the full database, suggesting that they are not linked to specific records.

The solution is clear. I can't do anything about how RM structures its GEDCOM file, so I will modify the plugin to tabulate data errors for manual checking later, but not abort the program when one occurs.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 22:53

Thanks for looking. And that seems like a fine solution. Thanks
Thanks,
Jackie

User avatar
tatewise
Megastar
Posts: 21528
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 05 May 2021 09:18

In FH there is the File > Validate... command and in RM7 there are File > Database Tools... such as Test database integrity that perhaps may repair those GEDCOM data anomalies.

FH also has the Compare/Sync Source Templates feature. Does RM7 have anything similar?

Jackie, have you any recollection of that CreditLine field possibly being a Citation field that has since become a Source field in the Source Template definition?
If Source Citations had been created using CreditLine while it was a Citation field, they might get left behind in the database after it gets changed to be a Source field.
The Test database integrity tool might correct that anomaly.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 05 May 2021 11:45

I ran all the database tools as soon as this came up. I've also run the database tools in Tom Holden's RMTrix which will sometimes catch things that RM misses.

RM7 automatically syncs the source template with the sources/citations as soon as you change them.

I used SQL Lite to examine (a copy of) the database. All the source tables look correct. The specific ones we are having issues with I also followed the links and they look correct. (Links may be the wrong term, I haven't done any SQL work in 30 years.)

It is possible the CreditLine was once a citation moved to a source, but I don't think it's likely I would do that. I always copy the RM templates and then modify. And I'd have no reason to have changed the CreditLine. And I checked the field in the source table to make sure CreditLine was a source.

All that said, this is an old, well used database. It originally came from TMG, then Legacy for awhile, then RM for a long while. So over the years, anything could have happened.

For me at least, Mark's plan to "modify the plugin to tabulate data errors for manual checking later" is a good solution. Even if I have a larger number of problem templates to correct manually, it will still save me a lot of work in changing my sources.
Thanks,
Jackie

User avatar
tatewise
Megastar
Posts: 21528
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 05 May 2021 11:53

That begs the question of where does the GEDCOM CreditLine citation field get its:
VALUE citing Pennsylvania Historical and Museum Commission, Harrisburg, Pennsylvania, Pennsylvania County Marriages, 1852-1973; County: Fulton; Year Range: 1947 - 1956; Roll Number: 549832

Does that text VALUE exist anywhere in the SQL?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 07 May 2021 22:31

All sorted now, thanks to Jackie kindly sending me a RM original of a large test file. It does appear that the RM GEDCOM export is flawed. However, a random sample of reported errors were all either source level data being duplicated in the citation or adding fields that were not present in the RM original. So far, we have found no examples of data being lost or corrupted. The large file gave around 400 such reports in 60k items of data being moved from UDF fields.

To recap, the plugin does 4 things consecutively
  1. Import the user-defined source templates from the RM GEDCOM file that FH discards. The resulting Source Template records are virtually identical to the RM originals.
  2. Reconstruct the built-in source templates as far as possible.
  3. Link RM sources back to their original templates, recreating as much as possible of the detailed structure that was lost when imported into FH.
  4. Move all the granular source data that are hidden in UDF fields back to their proper structure in the templated sources.
When the plugin encounters a data anomaly, the offending data are left as UDF fields and a Note record created to record the anomaly. This has details of exactly which template, source, fact, field, and individual/family it relates to, with clickable links for easy checking. If the attached Query is used to format the Records Window columns, you can view a spreadsheet-like table of all the anomalies that can be sorted on any column.

Over to the ex-RM users to try it out if you are interested. Test on a copy of your project first. It doesn’t matter if you have updated your project since importing from RM. The only things it uses the GEDCOM for are the template definitions. No old data are imported, and no project data are amended or deleted. It only moves UDF data back to their proper place.
Attachments
RM Source Import Error Records.fhq
(1 KiB) Downloaded 2 times
Import RootsMagic 7 Templated Sources (0.6).fh_lua
(26.51 KiB) Downloaded 5 times
Mark Draper

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 08 May 2021 09:21

As a wet weekend experiment, I have imported the test database into RM8 and exported from there. As far as I can tell, the results were identical. Good news - the plugin works on an RM8 export as well with no changes. Bad news - the data anomalies have not been fixed.
Mark Draper

Post Reply