* Export GEDCOM File - TNG - extra notes

[harold]

Export GEDCOM File 5.1
FH7 7.0.18
TNG 13.1.2 and 14.0

For some time I have noticed that when I export from FH and then import into TNG that literally hundreds of blank notes are created. A small SQL update in the TNG database usually takes care of them. I finally decided to investigate.

I export to TNG with the following options set.

Screenshot 2023-01-17 121819.jpg (166.5 KiB) Viewed 1098 times

I discovered that with these settings, every source which is based on a template has a reference to a note which does not exist in the GEDCOM. Since the note references seem to remain consistent among sources sharing the same template, I suspect they are artefacts which would be used to point to template descriptions if I chose "Move to Note Record" for the Source Template Record.

This is what I am seeing in the GEDCOM for a templated source I added to the Sample Project

0 @S41@ SOUR
1 NOTE @N3@
1 REPO @R4@
1 TITL Birth certificate of Anthony Munro, 12 December 1959, Collection of Nigel Munro.
1 ABBR Birth certificate of Anthony Munro, 12 December 1959
1 NOTE Type: Birth
2 CONT Region: Canada
2 CONT Principal: Anthony MUNRO
2 CONT Date: 12 DEC 1959
2 CONT Location: Orlando, Florida, USA
2 CONT Reference: Cert A123D4
2 CONT Repository: Collection of Nigel Munro
2 CONT Collection: special births
2 CONT Url: www.craswell.ca
1 CHAN
2 DATE 17 JAN 2023
3 TIME 12:01:50

A note record for the note @N3@ does not exist in the GEDCOM.

The issue with this comes when I import into TNG. It appears to create a place holder for the note @N3@ when it encounters the reference on import. The placeholder contains a blank note. Because it never finds an actual note definition it remains blank but is referenced by the source(s), hundreds of them in my case.

This is not a show stopper for me because I can run a simple update to clean them up after I do an import, but I wonder if it is an issue for others. Is there something I need to change in my settings to avoid this?

Harold

[tatewise]

Sorry, that is an error that has crept in somehow and will be fixed in the next Plugin version.

In the meantime, if you use the Move to Note Record setting instead of Reformat Metafields then the Note records are correctly created.

[harold]

Thanks for getting back to me and confirming my suspicions. I am only partially fluent in GEDCOM so I often second guess my interpretations of what I am seeing

In the meantime, if you use the Move to Note Record setting instead of Reformat Metafields then the Note records are correctly created.

I don't like to do that. When I do, I end up with a note containing the complete template definition, displayed with each source on my database. So in the meantime I will run the SQL updates on the TNG database after I do the import.

Thanks
Harold

[tatewise]

Well, if you use the Reformat Metafields option after I fix the problem you will get an identical set of Notes.

What about the Remove entirely option, which will remove those Notes entirely leaving just the Source record.

[harold]

When I use the Reformat Metafields option this is what I get in the Source window on TNG.

Reformat Screenshot 2023-01-17 172842.jpg (99.3 KiB) Viewed 1044 times

When I use the Move to Note Record option this is what I get in the Source window on TNG.

Move Screenshot 2023-01-17 173035.jpg (136.35 KiB) Viewed 1044 times

The note containing the reformatted Metafields is found at the very bottom of the second window after all of the other information. The information provided by the Reformat Metafields option provides added value to my website users. I want to make that data available but I may not want to force it all into the source title field. The data shown in the second screenshot just adds meaningless clutter to my website users. That is why I do not want to use the Move to Record Note option.

Perhaps after your fix is completed this will change.

I am also noticing what appear to be some anomalies (inconsistencies?) with how template fields are handled in citations as opposed to how they are handled in sources. I am as yet unable to formulate an coherent explanation of what I am observing. I may need to ruminate some more before doing that.

Harold

[tatewise]

After I have fixed the Reformat Metafields option, the Notes will be identical to the Move to Note Record option.
That is explained in the Help & Advice page for the Extra Options tab: https://pluginstore.family-historian.co ... ptions-tab

Templated source citation conversion is governed by the Source Template 1 _SRCT: setting.

• Move to Note Record ~ Each Source Template becomes a linked Note record. Each Source & Citation metafield becomes a labelled local Note entry, and any Source record Repository metafield also becomes a generic Repository link.
• Remove entirely ~ Each Source Template and each Source & Citation metafield are deleted completely.
• Keep Source Template ~ Each Source Template and each Source & Citation metafield are retained unaltered.
• Reformat Metafields ~ The Move to Note Record conversion above applies. Each Source generic Title is moved to the generic Short Title and the Footnote plain text becomes the generic Title.

The only difference between the two options you used is the content of the Title and Short Title fields of the Source record.

I suggested using the Remove entirely option. Does that remove too much for you?

[harold]

OK so now that finally makes sense for me. I have always found that section somewhat cryptic. In my experience it never actually worked that way so I thought I was somehow misinterpreting what I was reading.

The Remove Entirely option does remove too much in my case.

Also, I am still unsure about how template fields are handled at the citation level. It seems that the _FIELD tags are not removed at that level when the Remove Entirely or Reformat Metafield option is selected. I thought they would have been converted into the PAGE tag based on footnote formatting. I am not seeing that happen, but I need to go back and try and sort through the Help and Advice page.

Thanks for your help
Harold

[tatewise]

A while ago I spotted the _FIELD tags were not removed from Citation-specific templated fields and corrected the script.
Recently, there has been a stream of discussions involving the Export Gedcom File plugin needing script changes.
So I have decided not to release a new version for the moment until things have calmed down.

I've attached an updated Export Gedcom File plugin Version 5.1.6 Date 18 Jan 2023 ZIP file.
It is a ZIP file because the raw Plugin is bigger than the Forum attachments limit.
So you must right-click the downloaded ZIP file and use Extract All...
Then double-click the extracted FH Plugin (Lua) Export Gedcom File.fh_lua file to auto-install it into FH.

It should fix the missing Notes, the Citation _FIELD error, and several other minor issues. So please give it a try.

What you are experiencing is some of the effects described in the FHUG Knowledge Base of using Templated Sources.
Exporting GEDCOM to other products poses a challenge because Templates are not GEDCOM 5.5/5.5.1 compatible.
Different users and different products need various levels of detail that I struggle to keep up with.

[harold]

Thanks for the update. I will try it out later today. My dog needs a walk soon, so first I have to go brave the freezing rain outside.

What you are experiencing is some of the effects described in the FHUG Knowledge Base of using Templated Sources.
Exporting GEDCOM to other products poses a challenge because Templates are not GEDCOM 5.5/5.5.1 compatible.
Different users and different products need various levels of detail that I struggle to keep up with.

I am very well aware of that now .

In my digging I have been looking at a couple of developments on the TNG side of things as well. Available on the TNG WIKI is a MOD called Citation Master written by Bill Herndon. Essentially it provides a mechanism to format your source citation using available fields in the source. Various formats can be specified based on the source Type.

TNG can import values with non-standard tags. So I have been doing a bit of experimenting.

Using the example I created in the Sample Project yesterday. I exported the GEDCOM using the Keep Source Template option. This is the test source I created.

0 @S41@ SOUR
1 _SRCT @T50@
1 _FIELD EN-TYPE
2 ENUM Birth
1 _FIELD EN-REGION
2 ENUM Canada
1 _FIELD NM-PRINCIPAL
2 NAME Anthony /Munro/
1 _FIELD DT-DATE
2 DATE 12 DEC 1959
1 _FIELD PL-LOCATION
2 PLAC Orlando, Florida, USA
1 _FIELD TX-REFERENCE
2 TEXT Cert A123D4
1 _FIELD RP-REPOSITORY
2 REPO @R4@
1 _FIELD TX-COLLECTION
2 TEXT special births
1 _FIELD UL-URL
2 URL www.craswell.ca
1 TITL Birth certificate of Anthony Munro, 12 December 1959
1 CHAN
2 DATE 17 JAN 2023
3 TIME 17:23:29
0 _PLAC Salisbury, Wiltshire, England
1 MAP
2 LATI N51.0737
2 LONG W1.79512
1 CHAN
2 DATE 22 AUG 2022
3 TIME 16:31:17

Then I played around converting all of the _FIELD tags to new non-standard tags as shown below.

0 @S41@ SOUR
1 _EN-TYPE Birth
1 _EN-REGION Canada
1 _NM-PRINCI Anthony Munro
1 DATE 12 DEC 1959
1 _PL-LOCATI Orlando, Florida, USA
1 _TX-REFERE Cert A123D4
1 REPO @R4@
1 _TX-COLLEC special births
1 _UL-URL www.craswell.ca
1 TITL Birth certificate of Anthony Munro, 12 December 1959
1 CHAN
2 DATE 17 JAN 2023
3 TIME 12:03:18

I used the standard DATE and REPO tags where appropriate. TNG will import those non-standard tags and allow them to be relabelled. When I load the source I have effectively imported a rudimentary templated source into TNG. The Citation Master mod allows me to use those fields to structure custom footnote formats based on the Source Type.

So if there were a mechanism to convert a GEDCOM to a state where the template _FIELD tags were converted to non-standard tags, then I could effectively import the templated data directly into TNG .

However, I immediately see two hurdles.

• How does one reliably convert the _FIELD labels into tags? I thing there is a 10 character limit on tag length (at least in TNG) .

• AFAIK TNG cannot import non-standard tags at the citation level, so templated sources which use citation level fields may need slightly different treatment to transfer effectively

And now it is a mix of freezing rain and snow outside, and my dog really needs a walk.

[harold]

I downloaded and tried the update Export Plugin v 5.1.6 which you supplied this morning. It is now handing the export of templates in exactly the way you documented in the Help and advice. That clears up my misunderstanding based on the results that I was seeing previously.

Help and Advice.jpg (72.87 KiB) Viewed 911 times

However it leaves me in a slightly worse situation than I was in before. I have come to rely on this part:

Each Source & Citation metafield becomes a labelled local Note entry, and any Source record Repository metafield also becomes a generic Repository link.

It is useful to ensure all source related data is included even if it is not included in the footnote layout. (URLs for example)

The fix has re-established this part which I I would like to remove from individual sources:

Each Source Template becomes a linked Note record.

I have not problem (for documentation purposes) if the Note record is created and loaded into the database as a stand alone note, but I do not want it linked to every source since this really clutters up each source record on the website.

What would work best for me is another option, similar to the Reformat Metafields option which does not add the links from the Source Template Note record to each individual source.

[tatewise]

Ok, let me think about it.

Part of the problem seems to be that when importing the GEDCOM Source records, TNG clumps the local Note holding the Templated Fields with the linked Note (Source Template) record, rather than keeping them as separate entities.

[harold]

You have a point.

However they are kept as separate entities in the database. They are mashed together on display using a (join) table which links individual notes to other entities (such as sources). In order to clean this up I would need to find each of the join table records and remove them. There would be close to 1000 in my case.

Theoretically I could write a series of SQL statements to remove them after a GEDCOM import. There would need to be one SQL statement for each template type. Unfortunately they would need to be checked and modified after every import into TNG because there is no guarantee the identifiers for the GEDCOM note record would be the same in each export from FH.

[tatewise]

Overnight I've had an idea that after a quick experiment seems to work.

Create a temporary Copy Project of your master Project.
In that copy, delete all Source Template records and run the Export Gedcom File plugin with the Move to Note Record option.
That produces the Source local Note of Field values that you like but there is no linked Note (Source Template) record.

The only drawback I've found is that the Reformat Metafields option is the same as the Move to Note Record option because there is no Source Template definition of the Footnote format.

I think a similar result can be achieved by using the File > Import/Export > Export > GEDCOM File... command with the Records to Exclude option ticked for Source Template records, and run the Export Gedcom File plugin on the exported GEDCOM.

[harold]

Excellent!

Amazing what solutions come to us in the middle of the night. My wife used to think I was losing it, when she would occasionally wake in the middle of the night to find me on my phone, furiously thumb typing a quick email for work the next day. I found if I did not do that I would spend the rest of the night trying to make sure I did not forget my idea before morning.

I tried you proposal here and it seems to work fine. It amounts to just a couple of added steps to the export process, so that is perfectly do-able.

I have come to regret jumping down the template rabbit hole without fully understanding its implications at the publication (export) end. I thought I had checked all scenarios but obviously not thoroughly enough. Ironically, that is something I always cautioned clients about.

This gives me a solution I can work with for the present. Perhaps some day TNG and the main players in the industry will collaborate enough to work out a common protocol for exporting and importing templated source citations.