* Standard v Custom GEDCOM

[jimlad68]

[This discussion continues from Plugin - Change Any Fact Tag - suggestion (11325).]

In my new enthusiasm for Attribute Facts I had forgotten that while Custom Events are standard GEDCOM, Custom Attributres are NON standard GEDCOM, or at best Gedcom Extensions which will most likely have problems with portability.

i.e. a custom Attribute shows like this in the GEDCOM file:

1 _ATTR testnewattribute
2 TYPE testnewattribute
2 NOTE note for testnewattribute

The underscore in _ATTR being the givaway that it is NON standard GEDCOM (Gedcom Extension)

This is well explained and discussed in these 2 Topics:
Facts and GED standard. (10934)
GEDCOM Between Various Programs (11326)

I also tried the standard OCCUpation attribute with a very long line (400 chars), this created a GEDCOM line much longer than the max permitted 255 and with no continuation line. I suspect that most software these days will accept longer lines?

So, as useful as they may be in FH, using Custom Attributes might not be a good idea in terms of data portability, just have to try and use the existing standard attributes for my purposes.

From the Gedcom Extension List
As well as the _ATTR tag, some others are data related and might cause problems with portability. I am certainly not against using Gedcom Extensions, they are obviously very useful for FH software for things like flags and name lists, but I suspect even that could be managed via Custom Events.

What concerns me is:
1. FH tries to be 100% GEDCOM, yet extension tags like _ATTR, _USED (Given name used), and _STAT (family record status), are unlikely to export well, if at all.
2. While entering data into FH, there is no indication that these items are NON Standard and could create export problems.

[tatewise]

Certainly it would be useful if FH had an option to warn when using non-standard user-defined custom tags.

The GEDCOM Specification says such new tags only apply to the exporting system:

NEW_TAG := {Size=1:15}
A user-defined tag that is contained in the GEDCOM current transmission. This tag must begin with
an underscore (_) and should only be interpreted in the context of the sending system.

Many of the FH custom tags are of little or no use outside FH.
A few have equivalent popular de facto but invalid GEDCOM tags used in other genealogy programs.
As you have identified, the most prevalent and problematic is the custom _ATTRibute tag.

However, as has been discussed elsewhere, many genealogy programs simply ignore the distinction between Events and Attributes and allow both to have Values on the same line as the tag.

FH could have an Export > GEDCOM File option to replace _ATTR with EVEN.
FH could have a matching feature on importing GEDCOM to replace EVEN with _ATTR when it has a Value after the tag.

FH already has a related Export > GEDCOM File option to replace _FILE with FILE in Multimedia, but disguised as Destination: Family Tree Maker.

There are some Plugins that help with those types of conversion that could easily be expanded to cope with more cases.

Perhaps it is time that the glossary:gedcom_extension_list|> Gedcom Extension List is updated to advise on how to handle these scenarios and identify the Plugins that can help.

[DavidNewton]

After experimenting with Import-Export of GEDCOM I decided that I will eventually remove all the custom attributes from my files and replace them with either standard attributes/events or custom events. Mike's Change any Fact Tag plugin makes this feasible.

I was not particularly happy about it at first but many of the things that I wanted to see were in fact available by using override templates for the facts tab listings. It does have the advantage of making data transference much easier.

David

[AdrianBruce]

I would certainly agree with the idea of a report / query that could be run to identify data that is encoded in "custom GEDCOM", a.k.a. extended-GEDCOM, i.e. standard GEDCOM in the sense that it uses the standard underscore notation, but otherwise lies outside GEDCOM 5.5.

It would need to be runnable at any time, not at the time of entering the data but afterwards.

But of course, as many of us know all too well, just because it's in the GEDCOM 5.5 manual, doesn't mean it will load up into our target software. There is, I suspect, no safe-subset of GEDCOM understood across the board, so while such a report would be a nice tool, it is not a complete answer to cross-program GEDCOM compatibility.

[jimlad68]

David, unless mike has changed his plugin, a quick reminder of the limitations of the Plugin Change Any Fact Tag from the precursor Topic of this:
Plugin - Change Any Fact Tag - suggestion (11325)

Event to Attribute no problem. The note stays as the same note, but with a blank "Attribute Value". This is good because as far as I can see the "Attribute Value" can only be a single line and possibly limited to the maximum GEDCOM file length (which I think is 256, but historically from the old days of DOS often defaults to a Fixed Block length of only 80. I found this out when transferring GEDCOM files to the old Family Tree Maker from TMG), so no loss of data, but the "Attribute Value" may need to be entered or moved from the note.

Attribute to Event. This works fine, the note becomes the note, but it puts the "Attribute Value" in a SOURce note. Again no loss of data, but viewing the SOURce note is not obvious link in the property box or a report.

N.B. Although there is a GEDCOM standard max line length of 255, FH seems to create much longer lines (my test was 400), you would need to check how this exported.

[tatewise]

Assuming you use FH as your 'master' database, I would use all its useful features, and only worry about converting the Exported GEDCOM for the particular tags the target genealogy program does not handle.
That kind of conversion is generally easy with a Plugin.
The Export Gedcom to TNG Plugin has been specifically written to be adapted to different tag conversions.

As has been discussed elsewhere, there are even standard GEDCOM tags that are not supported by some genealogy programs.

I suspect that a Plugin to report all custom GEDCOM tags is perfectly feasible but if you take a look at the glossary:gedcom_extension_list|> Gedcom Extension List you will see that most of them are unavoidable and used extensively.

[DavidNewton]

Jim

Yes I was aware of the 'value' problem when converting attributes to events, I should have added that when I convert a custom attribute to an event I have a simple plugin to move all the attribute values to the beginning of the attribute notes first. (Which are usually empty anyway)

David

[jimlad68]

David, that sounds a useful plugin, also a good example, any chance of you posting it. I'm sure many might be in the same boat.
Jim

[tatewise]

That would seem redundant according to GEDCOM Between Various Programs (11326), which indicates that many programs quite happily accept EVENts with text on the same line, despite being invalid GEDCOM.

So I would continue to use _ATTRibutes and then simply replace _ATTR with EVEN in the exported GEDCOM file.
( It is an easy text editor global replacement of 1 _ATTR with 1 EVEN )
This is exactly what the Export Gedcom to TNG Plugin does.

[tatewise]

I am tempted to adapt the Export Gedcom to TNG Plugin into a general purpose Export Gedcom Data Plugin.

It could incorporate all the tag translation options discussed recently, not only for non-standard custom tags, but also standard tags that are not supported by some other programs.

There would be a preset combination of options for each popular program such as TNG, FTM, RootsMagic, Legacy, Gramps, etc.

Most importantly there would be a Strict preset combination of options that translated all non-standard custom tags into standard tags.

Once any preset has been chosen, the user could use an Advanced tab to customise the options where necessary.

So for example, the _ATTRibute tag would always convert to the EVENt tag, but an option would allow its value to either be retained on the same line or moved to its local NOTE field.

[jimlad68]

Mike, that sounds an excellent idea. Something that ideally should be built into the product, but the advantage of plugins is that they can be amended "on the fly" without a program update.

I suspect that was partially done by FH with their Export to Family Tree Maker and Strict GEDCOM options, but what a more comprehensive approach would mean is that we could use more features of FH without fear of losing data on export. This approach would also allow FH to develop more usability features that would normally be very bad for portability.

Not sure what you are envisioning, but I suppose even Flags (living, private, custom), could be exported as EVENts.

Just a note on your comment

.....indicates that many programs quite happily accept EVENts with text on the same line, despite being invalid GEDCOM.

from my tests I think Legacy and RootsMagic were OK, but TMG (now obsolete of course) and Rootsweb trees had problems. So your comment

So for example, the _ATTRibute tag would always convert to the EVENt tag, but an option would allow its value to either be retained on the same line or moved to its local NOTE field.

would make much sense, I would just amend that from ".... moved to its local NOTE field...." to "...moved to its local NOTE field, OR ADDED AS A NEW LINE TO THE TOP OF THE EXISTING LOCAL NOTE FIELD"

[tatewise]

Yes, when I said "moved to its local NOTE field" that would include inserting into an exiting NOTE.
Alternatively, it could possibly use a second instance of the NOTE field.
In any case, such moved text would be prefixed with a label such as Attribute Value: so it is easily identified.

The glossary:gedcom_extension_list|> Gedcom Extension List is gradually being updated to suggest all these possibilities.

[DavidNewton]

There has been quite a bit of activity on this thread today and this post is in response to a request from Jimlad68 last night to post a plugin. I have to say that the plugin that I used is very much manual intervention and requires inserting the Data Reference of the attribute into the plugin. I am enclosing the code but with a warning that unless you have a custom attribute labelled Name Change it will do nothing

Code: Select all

pi=fhNewItemPtr()
pic=fhNewItemPtr()
pim=fhNewItemPtr()
pi:MoveToFirstRecord('INDI')
while pi:IsNotNull() do
	pic:MoveTo(pi,'~._ATTR-NAME_CHANGE')
	while pic:IsNotNull() do
		strvalue=fhGetValueAsText(pic)
		if strvalue~=nil then
			pim:MoveTo(pic,'~.NOTE2')
				if pim:IsNull() then pim=fhCreateItem('NOTE2',pic) end
			notetext=fhGetValueAsText(pim)
			fhSetValueAsText(pim,strvalue..'. '..notetext)
			fhSetValueAsText(pic,'')
		end
	pic:MoveNext('SAME_TAG')
	end
pi:MoveNext()
end

To use this you need to alter the data reference in line 5 to match what you want. The value is inserted at the front of a note followed by a full stop and a space and any existing note is moved to accommodate it.

David

[tatewise]

I have a prototype Plugin V1.0 on my SkyDrive at Export Gedcom File.

It is still a 'Work in Progress' evolution of the Export Gedcom to TNG Plugin but stable enough for general use.
It cannot damage your Project as it only reads the source Gedcom file, and creates exported Gedcom files with very different filenames.

It incorporates the feedback on how various genealogy programs handle Gedcom files.
From the drop-down option at the bottom choose the desired target program.
Alternatively, just choose Strict Standard or Strict Minimum which both produce Standard Gedcom 5.5, but Strict Minimum removes custom features such as Named Lists and File Root whereas Strict Standard converts them to standard Note Records, etc.

All the custom FH features, and those not supported by specific programs, are converted to standard Note Records, local Notes, or user-defined Events. These user-defined EVENts, when derived from custom _ATTRibutes, may have values on the same line, or moved to their local Note depending on what the target program accepts.

You can also choose alternative export character encodings, but there is little difference unless you use accented and other exotic characters.

Feedback is welcomed on conversion problems, features not handled, or performance on large databases.

Later versions will improve the user interface, add advanced options to control individual feature conversions, and add online Help pages.

[jimlad68]

That looks good Mike, when my database is in better order I will try Family Tree Maker and it will be interesting to see if it will make a better fit to Ancestry.com trees, than Ancestrys own FTM, and Rootsweb Trees which usually takes nearly anything, somehow anyway.

[tatewise]

I have posted Plugin V1.1 dated 16 Aug 2014 on my SkyDrive at Export Gedcom File.

This is close to being complete, with bug fixes, online help pages, etc.

It just needs you guys to check its output GEDCOM is acceptable to all the destination programs.

[DavidNewton]

Mike

I used your new version to export to RootsMagic and came across an oddity which I have been trying to understand. Rootsmagic seems to import sources in a fairly standard fashion using the short title in the list of sources. However, scrolling down that list I found two sources that were new to me.

After some investigation I discovered that they were in fact source notes arising from changing a fact tag from a custom attribute to a standard event (Probate). The note was tagged SOUR and RM converted the text of the note into a source title and added it to the source list. This prompted me to check Legacy and FTM. I found that Legacy did exactly the same as RM but FTM (different as always) converted the note into a 'free floating' (i.e., no source) citation.

David

[tatewise]

Thank you for the feedback, that prompts a whole set of questions.

Does RootsMagic only handle Source Short Titles (ABBR) or does it also handle Source long Titles (TITL)?

Is it possible that they are all simply displaying the Source Notes in their own style?
If you export the GEDCOM from each of them, do they remain as Source Notes or become fully cited Source Records?

I know it may be too late now, but if you had retained your custom Attribute (_ATTR) the Export Gedcom File Plugin would convert it to a custom EVENt with the option of leaving the value attached or moved to a local Note.

You could use Change Any Fact Tag to change back, but each Source Note will have to be moved to the Attribute value by hand.

[DavidNewton]

Mike

In Rootsmagic the exported GEDCOM contains source fields ABBR, TITL, REFN, TEXT NOTE, OBJE.

Regarding the source Notes: RM and Legacy on GEDCOM export created a new source record with Title and Short Title equal to the text of the source note.
FTM2014: the GEDCOM export of the free floating citation was, as far as I can tell, a simple replica of the import. No new source record was created. This highlights one of the features of FTM. It is possible to add a source citation for a fact without linking it to any source record.

David

Added in Edit. Gramps also created source (and citation) records for the source notes using the text as a Title and nothing for the short title.

[tatewise]

Perhaps for RootsMagic, Legacy and Gramps any Source Notes should be converted to local Notes.

[DavidNewton]

Mike

I think you are correct. I only have a tiny number of source notes in my current working file but I will have a significantly larger number in other files and if each note is going to generate a new source record on export/import then perhaps action could be considered.

David

[tatewise]

On reflection, I wonder if it might be better for the Plugin to convert each Source Note into a dummy Source Record with a Title that starts with the label Source Note: followed by the text of the Source Note.

This is a closer approximation than using local Notes, retains the correct hierarchical tag position, avoids overloading already potentially 'busy' local Notes, and matches closely what RootsMagic, Legacy and Gramps already do, but with the benefit that the Title label identifies where the Source Record comes from.

[tatewise]

I have posted Plugin V1.1 dated 20 Aug 2014 on my SkyDrive at Export Gedcom File.

This should be slightly faster on large databases, adds a few more conversion Rules, and creates a Result Set of the Rules used and which Records were converted.

There is an Extra Options tab to allow the Rules to be customised for each target program format.

This version creates a dummy Source Record for each Source Note for RootsMagic, Legacy and Gramps as discussed above.