* Template issue when exporting back to RM8

[MFriend]

Hi Folks:
I'm looking to use RM 8 as a program to export to in order to upload to Ancestry and Familysearch.
The Project I'm working with was a direct import from RM 8. When imported into FH 7 it imported the citations as a "Free-Form" template. The problem is that when doing a gedcom export using the "Export Gedcom" plugin, it exports the citation section (the .Page section after imported into FH7) and merges it with the Citation text (which in RM8 is the Reports section:






So I guess my question would be: what settings should I have in the export plugin that might work to insert the .page section (which really is just the citation section) back into the .page section in RM8? (and not have it merge with the transcriptions, etc that goes into the Citation Text (Citation Text from Source) section?

If you need more pics or a better explanation please let me know...

Matthew

[Mark1834]

Matthew, Mike is the expert on his Export Gedcom plugin, but its main design philosophy is preservation of FH data in a useable format rather than full two-way compatibility of specific features such as Source Templates.

It's worth exploring why you want to use RM as an intermediary between FH and Ancestry or FamilySearch. The more transfer between apps there is, the more potential there is for data corruption or loss.

There are a number of ways of sharing data between apps, so if we understand better what your desired end objective is we can tailor advice accordingly. Be aware though that FH has fewer direct sync options than either RM or FTM.

[kfunk_ia]

It's worth exploring why you want to use RM as an intermediary between FH and Ancestry or FamilySearch. The more transfer between apps there is, the more potential there is for data corruption or loss.

Be aware though that FH has fewer direct sync options than either RM or FTM.

It would seem as if you answered you first statement with the last. The need to go back to RM is precisely because FH doesn't have options to upload data to Ancestry.

[tatewise]

Mark has said much the same as I would say regarding advanced features such as source templates and associated fields.
The stated objectives of my Export Gedcom File plugin say it does not fully reproduce all destination product features.

The philosophy is that FH holds the master database, and the exports are secondary copies for other products, perhaps to obtain online hints in Ancestry, FindMyPast, MyHeritage, etc, or to carry on a tablet when visiting record offices, churches, relatives, etc. Thus it is not critical that every tiny detail migrates from FH to the secondary product database. Migrating every detail is often not possible anyway, due to the product GEDCOM dialects in use.

In most cases, such non-GEDCOM features cannot be represented in the exported GEDCOM file anyway.
Such features would need a direct export feature to complement the direct import feature in FH.

If the objective is synchronisation with Ancestry then we need to understand how templates and fields are involved.
Ancestry does not support source templates and associated fields, so I believe they would get ignored.

Could whoever is interested explain their objectives for synchronising with Ancestry.
If it is primarily for preserving and obtaining 'hints' then my guess is that source templates and citations are irrelevant.
If Ancestry 'hints' are of no importance then export directly from FH to Ancestry is probably sufficient.
After exporting to RM and synchronising with Ancestry, what are the problems?

[ColeValleyGirl]

As somebody who exports to Ancestry to create 'cousin bait' as well as get DNA thrulines, and who keeps my tree public, I want to include sources and citations because I want people looking at my tree to understand that there is evidence supporting it. I ignore unsourced trees when I find them on Ancestry, and know others do the same.

[tatewise]

Perhaps I should have made it clearer that I thought templated source citations are irrelevant.
Those are still transformed to standard GEDCOM source citation notes by the plugin and Export > GEDCOM File command.

Helen, what export to Ancestry technique do you employ. That may help the OP to understand what works.

[ColeValleyGirl]

Perhaps I should have made it clearer that I thought templated source citations are irrelevant.

Strictly speaking, only the templates are irrelevant -- the sources created from the templates are very relevant.

I've used the Export Gedcom File plugin in the past, but not since migrating to FH7 as I haven't done many updates to my tree. I'm intending to experiment with the approach here: Synchronizing Your Tree With Ancestry to Exploit Hints but am not in a position to recommend one over the other yet.

[MFriend]

Like Helen I want to be able to export data from FH 7 that includes the sources and citations so others whether on Ancestry or FamilySearch can have access to them. There have been several times where every tree I have found on Ancestry has the wrong data on an Ancestor. After a lot of hard work I found sources that ‘proved’ a different set of parents, etc.

I am much closer to the end of my life than the beginning and it is important to me that I have my research available on the internet at places like Ancestry/Familysearch where it may benefit others someday when I’m gone (I so far have found no family members who want to do genealogy and my close family only puts up with it lol).

Background:
Anyways, the family line I’ve been working on had a lot of messed up sources/citations especially those that originally were ‘automatically’ created on Ancestry. I had imported the tree initially into FH7 from FTM 2019 after trying to fix the citations. Then in FH 7 I renamed and fixed the media. My last step was to try to get the places in the database standardized. The places were a real mess and while I love FH 7, it is quicker to update and standardize places in RM 8. So I exported a gedcom from FH 7 into RM 8. I never used any templates in FH 7, just the generic source.
In RM 8 I fixed and tidied up the places. THEN came the (it seems) fatal mistake. Instead of exporting a gedcom from RM 8, I did a direct import from RM 8 into FH 7. (Remember, I had not entered ANY citations in RM 8. This was just the data exported from FH 7 that I had fixed the place data).

With the direct import, FH 7 saw the citations and as a result the citation field (what FH 7 calls the Where Within Source field) as a template. I didn’t perceive that as a problem since FH 7 called It a “free-form-form templated which I mistakenly thought was the same thing in practice as a generic source. I then spent a solid 70 hours over the last couple weeks fixing things, correcting some missed citations etc.

Current Issue:
Now I have the citations, pictures, places the best they have ever been in this file, but now I can’t export it. Not just to RM 8. I can’t export that a gedcom to FTM 2019 either that is readable.

-- Using the FTM 2019 export settings (and importing into FTM), the citations are missing for 80% of them. Some it seems show up in the weblink section (?) but none show in the FTM citation section.
-- In RM 8 I was incorrect. They are not being all merged into what RM 8 calls the “Research Note” section (the equivalent in FH 7 being the citation “Text from source” section. 95% it seems are just ‘dropped’ or lost. Here is an example:


Other than the transcribed text and weblink, the main citation is entirely gone ( I eventually found them in an import error log created by RM).

I never would have done a direct import of RM into FH 7 if I had known it would cause FH 7 to put all the citations into RM 8 templates and make export impossible (I’m hoping there is a way which is why I wrote this initially).
In case it is helpful, RM 8 has an import log that lists the import errors. Here is a short example (there are hundreds and hundreds in the RM8 import log which I can upload if needed. FTM gave no errors it seems that it saved in a log):



I appreciate how helpful all of you are on this forum. I'm sorry if it seems like I'm griping. You all have been wonderful with helping me and others here on the forum and I appreciate it very much.

Matthew

[tatewise]

Matthew, thank you for that background. It helps a great deal.

I don't understand why a standard generic Citation field became a Template based Page field in RM8.
It should be relatively easy, perhaps with a plugin, to convert them back in FH and get rid of the Source Templates.

There appear to be some possible misunderstandings in your description.

What FH 7 calls the Where Within Source field is nothing to do with templates.
It is the standard generic GEDCOM ...SOUR.PAGE field.
Your Citation Window screenshot is somewhat confusing because it uses non-standard labels for the boxes.
Base on your earlier Customize Citation Window screenshot it is clear that:
• Page: is a Templated Citation-specific Field ~ Citation-specific: <Template Fields>
Citation: is the standard generic Citation-specific: Where Within Source field
Citation Text: is the standard generic Citation-specific: Text from Source field
Web Links: is the standard generic Citation-specific: Note field

The Export Gedcom File plugin (not the File > Import/Export > Export > GEDCOM File... command) should retain the standard generic Citation fields unaltered. The templated • Page: field text should be moved to the standard generic Note field. So the Citations should be retained in both FTM and RM8 although some details may be in a different field.

It would be useful to see snippets of exported GEDCOM equivalent to your screenshots of Citations involving • Page:.

[MFriend]

Thank you for helping Mike

Yes, before exporting to RM so I could fix the places, The citation info was in the "Citation-specific: Where Within Source" field (which I just renamed citation for the label). Before when I've exported using a gedcom from Rm to FH, the fields stayed correct. Only with the direct import (and not importing using a gedcom) into FH did it assign it to a template field.

Here are three examples of the Page field:



I've also zipped and uploaded a partial gedcom: https://familytreehelper.us/example_gedcom.zip

If you need more info let me know

Matthew

[tatewise]

Thank you. That confirms what I said. The templated Page field gets added to the Citation local Note with the Weblinks.
So the Page field Citation details are all present, just in a different field.
How does the Citation Note appear in RM8 and how does it sync with Ancestry?
The fact should still have a Citation. Does it matter that the details are not exactly the same as in FH?

[MFriend]

When I export from FH7 to RM8, the citation field is not importing into the RM file (only place I can find it is in the import error log).

So I have good news and bad i guess. First, I did an export of the RM 8 original database (after I had cleaned the places and the version of RM 8 tree that was imported into FH7). I exported a gedcom and imported that into FH7 (instead of doing a direct import) and the citation field mapped correctly and did NOT import as a template. Below on the left is what the original citation looks like, the middle column is FH7 after doing a gedcom import, and the far right column is the FH7 direct import results. Doing a gedcom import gives much better citation results into fh7 but loses some data like place details.



I was thinking we could fix this, but now I think I'm just going to have to accept the loss of dozens of hours of work this week and last. I've discovered another issue with the direct import that fixing which field the citation goes into won't fix.

In this testing I also discovered another issue with doing a direct import (I am so mad at myself for missing the problem till I had put so much work into it).

Background:
The citations in the original RM are linked to more than one fact (so the 1920 census below is linked to name, birth, residence. During import into FH7 the citations are changed so there is a separate citation for each of those three facts (not a problem, I expected it). The problem though is that the media links attached to the citation are NOT kept for each citation. From looking at several dozen examples, it seems it always keeps the link to the media for a citation on the primary name but very, very, rarely to citations on other facts (so in this example the copy of the 1920 census on the name has media, but the citation copies on birth and residence have no media linked/attached). Interestingly though, for the gedcom I imported, it kept the proper links to each citation.

Here is an example:
Original in RM on left: With a Gedcom import into FH7 Center: With the direct import into FH7 Right:
2 media attached .... Both media attached with gedcom import .... Direct = No media attached



I think it will be quicker for me to just use the gedcom export I've created and re-do the work I've done this week rather than what would be even more work I think trying to re-attached all the media to the citations. Here is an example of the death fact compared. Gedcom export on the left, Direct import into FH7 on the right. You'll see most of the other listed facts are also missing media (the media still being attached to the project, but not attached to all the citations).
Example:



So I appreciate your time you've spent looking at this, but I have to start this last import over again anyways

Matthew

[tatewise]

Is it possible that something has changed in RM8 since RM7?
Using RM7 and File > Import > GEDCOM (other programs) gives the following Source Citation data.
From the Edit Person dialogue, select any Fact, and the Sources button centre right identifies the number of citations.
Click that Sources button to display the Citation Manager dialogue where each Source Name record can be selected.
The Details column on the right holds the FH Where Within Source text.
The Source text / comments box holds all the FH Text from Source transcription, Source record labelled templated fields, and optionally the Source Template details.
The Research text / comments box holds the FH Citation local Note, and Citation-specific labelled templated fields (which is where your Page: field should appear).
I could not find any error logs.

[Mark1834]

Remember that there are significant differences between sources in FH and RM:

• FH sources can be generic or templated. In RM, ALL sources are templated. Even the "Free-Form" source is a template.
• FH and RM source templates are NOT compatible with each other. A direct import of RM templates into FH preserves all user data, but not UI details such as user prompt. FH templates cannot be exported into RM.
• FH is optimised for use with "split" sources. RM makes much more extensive use of "lumped" source templates.
• In FH, a common citation to multiple facts from a lumped source creates multiple independent citations. RM8 (but not RM7) has the option of a "shared citation", similar to FTM.
• FH does not permit citation-specific bibliography entries, while RM does.

There is an important principle here that users need to remember. No two family history apps are fully compatible with each other. DO NOT move your database backwards and forwards between apps and expect it to survive unchanged. Select ONE app as your master copy, and create (imperfect) copies in other apps to exploit particular features (Ancestry sync, website generation, etc). Regard this as strictly a ONE-WAY copy, and create new copies when the tree needs to be updated.

Moving a database around to exploit better source handling in FH, better place handling in RM, and better Ancestry sync in FTM doesn't work, as you have found out to your cost. Choose ONE app as your master, and work within its limitations.

[tatewise]

Yes, that is effectively what the introduction to my Export Gedcom File plugin says and explains the problems the plugin would have to overcome to export template based structures from FH for import into RM and other products.

As I reported yesterday, the plugin exported GEDCOM imports all the Source Citation details into RM7 but with the templated details in different fields. That should allow sync with Ancestry including reasonable Source Citations.

Is the GEDCOM import into RM8 different in some way?

[Mark1834]

Not compared them in detail, but perhaps RM8 uses the shared citation that RM7 doesn't have? It's identical in my Ancestry plugin, but that doesn't include sources.

In principle, it would be possible to do a near-perfect exchange of templated sources and their templates via GEDCOM, as the fundamental structures are the same - only the tag names differ. It would essentially be my old Import RM Templated Sources plugin in reverse. Whether it is worth the effort is another question of course - I'm certainly not offering to write it...

[tatewise]

I have discovered that I can install RM8 alongside RM7.

I imported the same plugin exported GEDCOM as before into RM8 with similar success but the data is located differently:
From a Pedigree View or Family View, double-click a person box to display their Name, Relatives, and Fact details.
Select any Fact to display its details in the righthand panes. Increase the height of the top pane so there is no scroll bar.
Under Fact Items the number of Sources (Citations) is shown, so click on the box to display them in the Sources pane.
Each Citation shows the Source record name with the FH Where Within Source text on the line below.
Click on the word Citation on the left of the top pane to display its Details in the lower pane.
Quality: Is the FH Assessment
Repository: holds the FH Repository record details.
Source ref: holds the FH Source record Custom Id.
Source text: holds the FH Source record Text from Source transcription.
Source comments: hold the FH Source record Note, labelled templated fields, etc, and optionally Source Template details.
Research notes: hold the FH Citation-specific Text from Source transcription.
Detail comments: hold the FH Citation local Note, and Citation-specific labelled templated fields (which is where your Page: field should appear).
I could not find any error logs.

So all the Source Citation details from FH were imported as expected and thus should sync with Ancestry.

In case it is not obvious, the GEDCOM file to use is the one that ends in RMT UTF8 produced by using the GEDCOM Destination: (RMT) Roots Magic Tree, not an ANC or FTM GEDCOM.

[Mark1834]

Yes, Team Buzbee have done a good job in enabling the two versions to coexist side by side on the same PC, and even be open simultaneously (Team Orde take note, but it would probably be difficult to retrofit into FH).

Under the hood, an RM8 database is just a superset of RM7, with some new tables or new fields to existing tables to exploit new features. SQL queries that use common tables generally run in either version, but there are one or two examples of changed field name or data format to trip up the unwary. I always test my various RM plugins in both versions.

[tatewise]

The point is that I don't understand why Matthew is having problems importing the GEDCOM file to sync with Ancestry.

[MFriend]

Hi Mike and Mark:
I've did a little more testing. I spend a good 20+ hours this weekend re-entering most of my data into the original RM 8 file that I had originally imported into FH 7 (I had entered that data into FH7).

Anyways, I am able to export a gedcom from RM 8 and import it into FH 7 without as far as I can tell any loss of data (the places, media links, media and sources/citations are all ok. I then exported a copy using Mike's wonderful export plugin back into a new RM 8 file and spent an hour comparing citations, places, media, and links. I was not able to find any problems. By exporting a gedcom and NOT doing a direct import, the citations are not imported as templates and hence export perfectly back into RM 8. I'm not a power user of RM so I don't have any custom facts, etc. and as far as I can tell FH 7 exports perfectly back to RM 8.

Now with the direct import that is another story. I tried a different family tree and experienced the same issues: Citations are imported as templates which will not correctly export to RM 8. Also, a big issue seems to be that media links are not imported correctly with the citations just like the other family tree file. It seems to me that when a citation is reused in RM 8 (say by running the citation deduplication), then when a direct import is done into FH 7 and the citation is copied to multiple facts, the citation link is only copied to the first fact and is missing from the remaining facts. In my case, that means that in both family trees I tested about 2/3rds of the citations are missing the associated media that was linked in RM 8. BUT as stated above, if a gedcom export from RM 8 is imported into FH 7, then the links are properly maintained.

My recommendation to others who want to import a tree from RM 8 into FH7 would be to do it through a gedcom export and not a direct import (though I guess if they have heavily customized data in RM 8 or used Templates in RM 8 maybe direct would be the way to go).

Anyways, I started the process of trying to fix and edit all the citations, media names, and places for my two main trees back about April 19th of this year. After two solid months of probably 7+ hours a day working at it I can say I'm happy with where my trees are now. They aren't perfect, but they are much, much better than before. I finally feel like I can start conducting new research and this time am planning on manually entering the data and media so I don't have to go through this again.

Matthew

[Mark1834]

My recommendation to others who want to import a tree from RM 8 into FH7 would be to do it through a gedcom export and not a direct import (though I guess if they have heavily customized data in RM 8 or used Templates in RM 8 maybe direct would be the way to go).

Sorry Matthew, I completely disagree with you. ALL sources in both RM7 and RM8 are templated, so how can somebody not use templates? A GEDCOM import loses all the detailed granularity of the source structure. Other RM features such as the ToDo list and Research Logs are discarded altogether. I should know - I spent a lot of time last year writing plugins to correct as many of the deficiencies of the GEDCOM input as I could!

There are a significant number of RM migrants on this forum now. This is the first I have heard of missing media citations, and I'm sure it would have come up before if it were a general issue. But let's hear from those who have had a more positive experience of the migration. Do you recognise any of these issues?

I will do some controlled experiments tomorrow to see if I can duplicate any of the media issues you are describing. If we can identify a specific problem, it will need reporting to CP.

[Mark1834]

I decided not to wait until tomorrow, as it was a simple test...

I created a test RM8 database containing 2 individuals. Each person had a birth, census, and occupation fact sourced from the same census entry, which had an attached media file. For one individual (Mary Jones), I created a source citation for the census fact, copied that citation, and pasted COPIES of that citation to the birth and occupation facts, which is the RM7 way of doing things, and similar to how FH handles multiple citations from a split source. For the other individual (John Smith), I followed the same sequence, but the copied citations were SHARED, which is a new feature in RM8.

Where the citations were COPIED, media were linked correctly in FH. Where the citations were SHARED, media were dropped from two of the three citations, as you reported.

It's probably dangerous to generalise from this one experiment, but it is possible that this has not been noticed before because most RM migrants have been RM7 users who don't like RM8, and perhaps their media would have come across correctly.

I will raise a ticket and forward my test database to CP tonight. This appears to be a significant bug in the RM direct import. I reiterate my advice NOT to resort to a GEDCOM import as a workaround, as this creates more problems than it solves, but clearly recent or intending RM8 migrants (particularly those with SHARED citations) should either hold off from migrating until this is resolved, or check their database carefully if they have migrated already.

[tatewise]

The missing Shared Citation Media problem is a side shoot of the original problem that will hopefully be fixed by CP.

Matthew, referring back to the original problem you said that the Export Gedcom File plugin generated GEDCOM when imported to RM8 contained various errors and omissions and you provided some evidence. As I explained over the weekend, I cannot see any such errors or omissions when importing to RM7 or RM8. Can we now focus on that please.

BTW: I agree with Mark that direct import from RM7/8 into FH V7 is far superior to GEDCOM.

[Mark1834]

Looking at the FH project GEDCOM, it looks like a simple matter of the link to the media file not being included when an RM8 shared citation is converted to multiple FH copies (FH does not support shared citations). Hopefully, it should be a straightforward fix, but let's see what CP say...

[Mark1834]

Reading the complete thread again, I suspect that Matthew may be getting confused between the RM "Free Form" source and FH Generic Sources. He said early on that I never would have done a direct import of RM into FH 7 if I had known it would cause FH 7 to put all the citations into RM 8 templates.

However, CP explain the difference in their guide to importing from RM: When you add a source citation in RootsMagic, you can choose to either use a source template or enter the source citation free-form. When you add a source citation in Family Historian, you can choose to either use a source template, or use a generic Source record. It might seem then that a free-form source in RootsMagic is the equivalent of a generic Source record in Family Historian. In fact, they are not equivalents. A generic Source record is not free-form. Instead it uses standard fields, for sources of all types. To support RootsMagic’s free-form source citations (i.e. source citations that you just type yourself) a special Source Template record called “Free-form”, is automatically created when you import into Family Historian, and all free-form Source record details become Source records that use the Free-form Source Template.

So in RM, "Free Form" sources are essentially sources with a blank template, but FH creates that template when they imported.

I have experimented with moving data around using just Free Form sources, and while the results weren't always very pretty, I didn't see any evidence of data corruption (unlike fully templated sources, where GEDCOM transfer destroys the detailed structure).