* GEDCOM Between Various Programs

[DavidNewton]

This discussion continues from Plugin - Change Any Fact Tag - suggestion (11325):

It will be interesting to see how other programs deal with Attribute vs Events.

As it happens I have been concerned about this as the eventual recipients of my files will have a variety of programmes. This afternoon I created a strict GEDCOM export of my personal file (1300-ish individuals) with no media and then tried to import the GEDCOM into the other programmes that I own. The results were interesting:

FTM 2014
The import log shows 18 errors all stating that ASSO Unsupported or Invalid. Sounds good, but, on examining the file more closely I discovered that all custom attributes had been lumped together as _ATTR with no indication of the type of attribute.

RootsMagic 6
RootsMagic imported the file without stating that there were any problems. However, I have imported before and it puts an import log alongside the imported file. (A text file with a LST extension) This file showed 2219 instance of 'Unknown Info' including all custom attributes, ASSO, AGE and a number of Update date and time records. On examining the file in RootsMagic all custom attributes had been discarded.

Legacy v8
After one abortive attempt, where Legacy went into an infinite loop of analysing the GEDCOM and required Task Manager to stop it, I re-exported the file from FH and there was no problem (Uh!). Legacy failed to recognise the Flags (Living etc) but did offer to map them to other tags before import. I didn't take the offer (to save time) but it would have been simple enough to map them. The import then proceeded with no further problem. On examining the file I found that all custom attributes had imported correctly and the Age fields had been put into the Event Note fields in the form 'Age at this event: 15y'

Obviously this is a single trial with one small GEDCOM but does show that it is generally unsafe to assume that a GEDCOM import will give you a file which matches the one you started with.

David

[tatewise]

It just goes to show that despite existing for nearly 20 years how poorly GEDCOM 5.5 has been implemented.

It is particularly sad that Standard tags such as ASSOciated and AGE are not supported.

It is not entirely surprising that Custom tags were rejected or mishandled.

It seems that Legacy v8 did a particularly good job apart from the AGE field.

The RootsMagic 6 problem with CHANge dates & times is tolerable, as those fields are optional.
It can be fixed with a Plugin I wrote recently.
See Remove last updated (CHANged) date from exported GEDCOM (11305).

A further test is to export GEDCOM from those programs, especially after using their custom features.

[DavidNewton]

Mike

I will have a look at exporting from these programs. Both Rootsmagic and Legacy support shared events and it will be interesting to see how they are disentangled. Unfortunately I have no recent files since I switched to FH but I will dig around.

I was particularly disappointed with Rootsmagic as that is one of the two (Legacy being the other) which provides a free limited edition. So I had another go but this time I preprocessed the GEDCOM using Notepad++ and simply replaced all occurrences of _ATTR with EVEN and that seems to have solved the custom attributes problem of course it may well have raised other less obvious problems. Another solution, not universally available, is that Rootsmagic can happily, and apparently correctly, import directly from Legacy, so a two-step process also does the trick.

But. I agree with you, it really should not be necessary.

David

[tatewise]

_ATTRibute is the FH Custom tag for Custom Attributes, whereas EVENt is the GEDCOM Standard tag for additional Facts.

However, EVEN does NOT support a Value on the same line in the way _ATTR and most Standard Attributes do.
i.e.
These are valid:
1 OCCU Labourer
1 _ATTR Any text value
2 TYPE Name of attribute
but this is NOT:
1 EVEN Any text value
2 TYPE Name of event

So how did RootsMagic handle that scenario?

[DavidNewton]

Mike

I am selecting one custom attribute entry from my file (and I know the census use is non-standard). I guess you will be somewhat surprised at the outcomes. In the GEDCOM produced by FH we get

Code: Select all

1 _ATTR Head of household
2 TYPE 1851 Census
2 DATE 30 MAR 1851
2 PLAC Sneinton, Nottinghamshire
2 ADDR Clerment Place, 
2 AGE 36y
2 SOUR @S207@

I then changed _ATTR to EVEN and imported into RM and have now exported from RM to a GEDCOM. This entry becomes

Code: Select all

1 EVEN Head of household
2 TYPE 1851 Census
2 DATE 30 MAR 1851
2 PLAC Sneinton, Nottinghamshire
2 ADDR Clerment Place,
2 SOUR @S207@

As you can see Rootsmagic deals with place and address in the same way as FH, ignores the Age field and attaches a value to the event. RM is not alone here. If I export the same file from Legacy I get

Code: Select all

1 EVEN Head of household
2 TYPE 1851 Census
2 DATE 30 Mar 1851
2 PLAC Sneinton, Nottinghamshire
2 NOTE Age at this event:36y
2 SOUR @S121@
2 ADDR Clerment Place,

Legacy does not have two part location fields but the address is preserved in a separate Event Address box. And finally the output from FTM 2014 for this event

Code: Select all

1 EVEN Head of household
2 TYPE _ATTR
2 DATE 30 MAR 1851
2 PLAC Sneinton, Nottinghamshire
2 NOTE @N14@
2 SOUR @S207@

This successfully hides the fact that a Census is involved, still applies a value. I looked up the Note references and we have

Code: Select all

0 @N14@ NOTE
1 CONC Address: Clerment Place,

Edit: I should have said only the RM import needed the change from _ATTR to EVEN. Legacy and FTM2014 are imports from the original FH GEDCOM file

[tatewise]

TNG has a similar problem of invalidly allowing EVENts to have Values.
See From TNG > FH5 > TNG (11226) especially from Mon Jul 07, 2014 onwards.

If you import any of those GEDCOM into FH it will correctly complain about Tags with invalid Values and discards entire Event:
EXCLUDED: invalid line : "1 EVEN With A Value"

[jimlad68]

interesting stuff. From my very limited GEDCOM from FH these were my observations which tie in with the above:

<> Legacy and Rootmagic.
I noticed that both of these did not separate Attributes and Events, all Events(Facts) could be either, by include/exclude the description field (Attribute Value).
For my simple file, but with many Facts and simple sources with text, both did surprisingly well. I too have heard from the TMG forum of people keeping a database on Legacy and duplicating it on Rootsmagic.

<> TMG - a disaster
- my sources were mostly lost and source notes got converted to Event-misc.
- Custom Attributes got converted to Event-misc without the "Attribute Value", although BIRTh etc were fine.
- The ages were added to the Memo (text) as Unknown GEDCOM info.

I was very surprised with this as with the flexible export options TMG gave me an excellent EXPORTed GEDCOM file (admittedly I had very basic Source detail), possibly with extra manipulation of the TMG import process better results may have been had, but I could not see anything obvious.


<>FTM 2014 - I didn't waste my time!!!

[DavidNewton]

Just for the record. I forgot that I had a copy of Gramps installed (v3.4.8-1) Importing from FH to Gramps and then exporting to a GEDCOM gives (with the event matching my earlier post)

Code: Select all

1 EVEN
2 TYPE 1851 Census
2 NOTE Description: Head of household
2 DATE 30 MAR 1851
2 PLAC Sneinton, Nottinghamshire
2 ADDR Clerment Place, 
3 ADR1 Clerment Place, 
2 AGE 36y
2 SOUR @S0207@

So Gramps seems to adhere to the GEDCOM standard here with no text on the same line as the EVEN, moves the 'value' to the Notes field and supports the age field. A quick check also reveals that ASSO is supported. I am not sure why there are two address fields.

I have to say that from this point of view I am finding Gramps impressive although there does some to be a bit of a learning curve with the interface.

David

[tatewise]

There are several optional subsidiary fields of the 2 ADDRess field:
3 ADR1 = Address line 1
3 ADR2 = Address line 2
3 CITY = City
3 STAT = State
etc...
It looks like Gramps may be automatically filling each sub-field from comma separated parts of the ADDRess field.

[jimlad68]

I might have been a little harsh on FTM 2014 before, if it is used a lot it is very viable in this "experiment". One sees reviews in "review sites" (often not very good) on about 10 different genealogy programs, but I have never seen any indication of how popular they are, how many sold, how many in use as a main database, probably very difficult to compile.

As well as sharing with other programs there are other places, as I have mentioned previously:

- Rootsweb Trees seems to take anything you throw at it.

- Ancestry Trees, admits that it does not take any note/memo type information, even from it's own FTM 2014, I have only got v slow internet at present, so I cannot check this, but I think it takes "attribute values" OK.

[tatewise]

Sometimes even FH is a little inconsistent.
e.g.
In the illustration above, if EVEN has invalid Value text on the same line, then FH EXCLUDES the whole Event.

Whereas, if a Standard Event such as BIRT or CENS has invalid Value text on the same line, then FH says INFO ONLY: Detected & fixed field format error (data moved to Note Field) and retains the Event, just as Gramps does.

[tatewise]

I have a prototype Plugin V1.0 on my SkyDrive at Export Gedcom File.

It is still a 'Work in Progress' evolution of the Export Gedcom to TNG Plugin but stable enough for general use.
It cannot damage your Project as it only reads the source Gedcom file, and creates exported Gedcom files with very different filenames.

It incorporates the feedback above on how various genealogy programs handle Gedcom files.
From the drop-down option at the bottom choose the desired target program.
Alternatively, just choose Strict Standard or Strict Minimum which both produce Standard Gedcom 5.5, but Strict Minimum removes custom features such as Named Lists and File Root whereas Strict Standard converts them to standard Note Records, etc.

All the custom FH features, and those not supported by specific programs, are converted to standard Note Records, local Notes, or user-defined Events. These user-defined EVENts, when derived from custom _ATTRibutes, may have values on the same line, or moved to their local Note depending on what the target program accepts.

You can also choose alternative export character encodings, but there is little difference unless you use accented and other exotic characters.

Feedback is welcomed on conversion problems, features not handled, or performance on large databases.

Later versions will improve the user interface, add advanced options to control individual feature conversions, and add online Help pages.

[DavidNewton]

Mike

I have been trying out your plugin with FTM 2014. The import into 2014 proceeds with no errors reported and hence this feedback is not so much about the plugin as a caution to anybody who intends to move data from FH to FTM2014.

My experience with FTM 2014 import of sources, and I have noted this behaviour in the past.

Data is discarded without warning by FTM. This is particularly important if you use method 1 sources. Exporting a method 1 source to GEDCOM will usually lead to a source record with SOUR tag at level 0 and then, using just the level 1 tags some subset of the following, depending on what fields have been filled in: AUTH, TITL, ABBR, PUBL, REPO, OBJE, REFN, TEXT, NOTE

Of these ABBR, REFN and TEXT are discarded by FTM2014.

A major effect of this for a Method 1 source is that the "Text from Source" is discarded. The effect of discarding ABBR in my file is that it is much harder to distinguish sources, particularly all the ones which have titles starting "Census of England & Wales taken on ..".

If you must pass on data for use in FTM2014 then the Text from Source should be moved into the Note field and you might also need to reconsider the titling of sources.

David

[tatewise]

Thank you David that is excellent feedback.
Armed with that info I can adjust the Plugin to automatically adjust those fields discarded by FTM.
I propose ABBR is move to TITL unless that already exists, when a labelled NOTE will be used.
REFN and TEXT will be moved to a labelled NOTE.

The whole point about this Plugin is that as we discover how other programs work, it can be updated to avoid users having to move any data within FH.

[DavidNewton]

Mike. Feedback on Legacy v8

Legacy is harder to test because there are numerous options and customizations available both with import and export. I simply accepted defaults as far as possible.

At the import Legacy noted that it did not recognize flag records and I was given the option of mapping them, creating custom events or nothing. I chose nothing and the flag details were then inserted into the General notes of the individuals involved. Other than that there were a few complaints in the log (5 I think) about unrecognized dates. Somewhat strangely 3 of the complaints were about quarter dates in family records and at a quick glance I couldn't see the problem. (My file, like most UK based, is littered with quarter dates and Legacy normally has no problems with them)

Source records. As far as I can tell all level 1 tags are faithfully reproduced and also reproduced in the (Legacy) exported GEDCOM.

There was one interesting phenomenon which took me a while to sort out. Legacy Note records are generally attached to other records. So the named list notes with no particular associated records could be a problem. Eventually I sorted out what was happening. Starting from the last Named list the full named list is attached as a note (together with its name) to everybody in the list unless they have already had a named list attached, in which case they are passed over. I would speculate (not tested) that this will fail if you do not choose the option on Legacy import to retain RINs

David

[tatewise]

The Plugin converts each FH Flag to a custom EVENt with TYPE set to Record Flag: <FlagName>.
Most programs (including FH) need custom EVENts to be defined/mapped for full functionality, so that is normal.
The same prompt will arise for any FH custom _ATTRibutes or EVENts that you have defined.

Strictly speaking there is no such thing as Quarter Dates in GEDCOM format.
They are usually represented as a Date Range between the first & last month of the quarter:
e.g BET JAN 1900 AND MAR 1900
If you post the problem Dates, maybe I can spot the problem.

The Plugin creates a Note Record for each Named List with all the record details.
As you spotted, it links each Record in the list to the Note Record.
BUT as you also spotted, if the Record is in more than one Named List/Note Record only one is linked.
This is an oversight in the Plugin, which I have already fixed for the next version.

The standard GEDCOM RIN tag for AUTOMATED_RECORD_ID is nothing to do with the above.
FH does not use RIN unless you manually use Add Miscellaneous > Add Automated Record Id in the All tab of an Individual/Family Property Box.

[DavidNewton]

Mike

I'm having a cloudy day! Why I should think that Legacy software was linking those notes I do not know, I have checked the log file and the unrecognized dates were all "" enclosed date phrases in my FH file.

My comment regarding quarter dates is because Legacy, like FH, will recognize quarter dates so long as the format is correct. In Legacy the dates are represented by the abbreviated last month of the quarter followed by Q, Thus Mar Q represents Q1 etc.

David

[tatewise]

That Legacy quarter DATE format is not GEDCOM compatible.
So presumably when Legacy imports dates like BET JAN 1900 AND MAR 1900 they stay in that format?
In what format does Legacy export its quarter dates to GEDCOM?

The Plugin could adjust the DATE fields for Legacy, but it is probably not necessary.

Referring back to FTM 2014, could you check how it imports Custom Id REFN fields for other Record types such as Individual, Family, Multimedia, etc.
I suspect it does not import any of them from a GEDCOM file.

[DavidNewton]

Mike

Regarding Legacy. I am certain that conversion of date ranges into quarter dates is not going to happen, far too many assumptions involved. On the export GEDCOM direction (from Legacy) one of the options that can be chosen is to not convert quarter dates to a range, but the default is to convert them. I have not tried to import a quarter date into Legacy since most of the software I have used either does not recognize quarter dates or automatically converts them to a range when exporting GEDCOM. As a quick test I used a quarter date in a small test file and exported it from Legacy with the option not to convert quarter dates. The date then came in the format Mar Q 1941. When I examined this using FH the date was written as a phrase "Mar Q 1941".

Regarding FTM. The custom Ids (REFN) are imported/exported without problem as far as I can see for Individual and Family records. I do not use multimedia custom Ids and when I added one to my file for a test, my copy of FTM 2014 went into what appears to be current default mode (according to various forums), i.e., not working. So further tests of FTM2014 cannot be conducted for now.

I also have a copy of Rootsmagic 6 installed and I will try to make time to test that also, perhaps later this evening.

David

[tatewise]

Thank you for all your work David.

I have updated the Plugin to V1.1 via the same OneDrive link above.
It uses my standard style user interface, adds some cursory help pages, and improves the conversion process by running faster and fixing any conversion deficiencies that have come to light. These include the Named List Note Record missing links and FTM 2014 lack of support for many Source Record tags discussed above.

Please use this version for any future tests.

[tatewise]

I have posted Plugin V1.1 dated 16 Aug 2014 on my SkyDrive at Export Gedcom File.

This is close to being complete, with bug fixes, online help pages, etc.

It just needs you guys to check its output GEDCOM is acceptable to all the destination programs.

[EDIT]
New version date 20 Aug 2014 now available.