* Ancestry, FTM, FH and workflow

[tatewise]

Those are useful suggestions that may be incorporated into the Export Gedcom File Plugin.

You say "It's disappointing indeed somewhat remiss of the Gedcom "specification" that data exchange is not straightforward." The GEDCOM Introduction says:

GEDCOM was developed by the Family History Department of The Church of Jesus Christ of Latterday Saints (LDS Church) to provide a flexible, uniform format for exchanging computerized genealogical data. GEDCOM is an acronym for GEnealogical Data Communication. Its purpose is to foster the sharing of genealogical information and the development of a wide range of inter-operable software products to assist genealogists, historians, and other researchers

It is not perfect, but few software programs implement it fully, and FTM is a prime example of why data exchange is not straightforward due to its flagrant misuse of the specification. That is not the fault of the GEDCOM specification! Have you tried complaining to FTM Support about its poor implementaion of GEDCOM?

[Nick-V]

Thanks Mike. I'm uncertain of the GEDCOM spec but does it allow Values against events ? FTM allows this but FH does not - I have no idea who is right or wrong and can't be bothered to fight...I want to address it as easily as possible. Further, my point was that Tags although used legitimately can used quite differently...does "Mr" belong in TITL or NPFX or does the spec not specify? I fancy the latter.

I am continuing to look and discover small useful matters in preparing data for the journey to FTM (like FAM, _STAT - local notes are ignored). We can liaise further if you wish to attack this subject.

One thing that would really help...in the image processing...providing the links from the INDI records to the full image and inserting a _PHOTO tag to create a default image...any chance?

[AdrianBruce]

Thanks for your efforts Nick. Interesting - and depressing to see how poor FTM's understanding of a simple specification like GEDCOM is. (Yes, it is simple to an IT person).

The reason things like promoting CAUS to level 1 depress me is that the level numbering system in GEDCOM is there for a purpose. An event (like the death event) starts with a level 1 and - as far as the GEDCOM spec'n is concerned, is terminated by a new level 1 (or even higher) that starts a new event, etc. So, as you probably now realise but others may not - promoting CAUS to level 1 means that, as far as "proper" GEDCOM is concerned, CAUS is a new thing and nothing to do with the death event.

I would myself quite like to use Ancestry's hints in an easier manner, hence my interest in this thread. I did wonder if there was an alternative route - you've probably thought this through ages ago. FH can do a direct import from some databases but, so far as I know, FTM isn't one of them. The FHUG Knowledge Base http://www.fhug.org.uk/wiki/wiki/doku.p ... tree_maker says that it was possible until FTM 2008, which encrypted the database. Then I thought about using an alternative man-in-the-middle such as RootsMagic - but RM can't talk to Ancestry because "Ancestry won't give us permission to access their API" according to a RM posting on Facebook. (Anyone see a theme developing here? Cynic? Moi?)

(In any case, the way I work is that the Source-Record is entered first, before the facts, with the "proof of identity" in the notes, and I'm not sure how synching hints would affect that - possibly it'd be all too tempting to do a half-job.)

[AdrianBruce]

Thanks Mike. I'm uncertain of the GEDCOM spec but does it allow Values against events ? FTM allows this but FH does not - I have no idea who is right or wrong and can't be bothered to fight...

For what's it's worth - no, it doesn't. Values are for attributes, not events.

Caveat - the only value acceptable for the standard events is "Y" allowed for some (all?) to show that the event has happened when no other data is known.

Apparently Ancestry's GEDCOM load misinterprets that, as many people are reported to have died at a place called "Y" in France. (Yes, it does exist - see https://en.wikipedia.org/wiki/Y,_Somme)

[tatewise]

Adrian's points are well made. Nick - You have not said if you have complained to FTM.

The GEDCOM spec does NOT allow values against Events, only against Attributes, and that is their only difference. However, many programs (excluding FH) do not make that distinction.

There is a major disctinction between NPFX and TITL.
NPFX is a sub-field of every instance of the NAME field and is where Mr should be held.
TITL is an Attribute just like any other Fact and can have a Date, Age, Place, Address, Note, and Narrative Sentence. It is intended for inherited or aquired or temporary titles like Lord Bath, Duchess of Norfolk, Sir Brokenbury, etc, similar to the OCCUpation Attribute, and certainly NOT Mr.

You have raised several cases of Custom FH tags such as _SENT and _STAT that should never appear in a GEDCOM from Export Gedcom File. Properly structured Custom tags always start with an underscore, whereas FTM just steals full GEDCOM tags such as CAUS or invents them such as EMAIL.

The plugin could create a _PHOTO tag, but how is the default image defined, is it the Pref 1 Media image?

[Nick-V]

Adrian and Mike

Thanks for clarifying my guess that Attribs have values and Events not. Thanks also for demonstrating where FTM may be abusing the spec.

I'm not into blame, I wish to be pragmatic and remain focussed on solutions irrespective of cause. "It is what it is"! If FTM wants to regard everything as an Attrib then so be it as long as we can translate back again.

I have not complained to FTM. Possibly if I was more knowledgeable and my findings more comprehensive I might tackle this but, frankly, I am quite certain they have already received complaints (yada yada) and I believe (even support partially) that they have made a deliberate decision to keep things as simple as possible - their major consideration being how to handle data in Ancestry and appeal to the masses.

That said, it must also be true that there is a massive customer base that would be most interested in using a far better product like FH and having a routine that enables both FTM (not Ancestry directly, we've rules that out APIs etc.) and FH to work hand in hand - even if some compromises have to be made. I am trying to progress a way to do this even if I need to constrain my use of FH, make tough decisions about fields used and do certain data changes manually.

At some point in my testing I noted a _PHOTO tag in the FTM export. When I created one it appeared to set the profile picture in FTM to the media record I had pointed at. I am suggesting we assume (reasonably) that the first media for an individual be the profile pic. I also suggest the record only be created if it does not already exist allowing overriding in FTM if round tip ever works. I will check again tonight and report back (edit here) if this is not true.

I recognise that round trip images have other issues...image names are prefixed by the export plugin...we don't want that happening every time!

[Nick-V]

BTW, FTM can import directly from the FH Gedcom, a Gedcom exported from FH and a Gedcom exported from the plugin...which is far better of course. However, it does not accept all the info as it should which is what we are trying to address. Indeed, there are differences between importing to Ancestry and FTM - perhaps not surprisingly! I'm focussing on FTM so that merge might be possible.

Syncing hints appears (brief test) to be fast to use and the data is kept well under the TEXT tag and indeed images of Census etc. are even imported back to FH. There is a difference in FTM is viewing images connected to a SOUR versus a citation which might be tackled tho !

[tatewise]

I am pleased FTM can import directly from a UTF-16 Project Gedcom as few programs do that.
Such direct FH Project Gedcom and File > Export > GEDCOM File Gedcom include all the FH Custom tags, whereas the Export Gedcom File Plugin excludes & converts them all, which is its main purpose, and I presume you have been using its (FTM) Family Tree Maker 2014 export mode.

[Nick-V]

Yes all three exports (preference for your plugin of course) and yes the FTM export to Gedcom...its own data is stored in a proprietary FTM database.

[Nick-V]

Mike...image processing:

Using existing terminology (I hope) I confirm that an export to FTM would benefit from:

1) Media Records and Absolute File Links for both Part and Full Frames (potentially multiple links to the full frames from each "face" pic to the "family" pic) - the BOTH option needs to employ Media records and create multiple links to full frames I guess.
2) _PHOTO record created as below for each IND referring to the first (part/IND) media record (if there isn't a _PHOTO record already)

For example:

Code: Select all

1 BURI
2 DATE 1996
2 PLAC Hounslow, London, England
2 ADDR New Brentford Cemetery, Sutton Lane
1 FAMC @F8@
1 FAMS @F3@
1 FAMS @F50@
1 _PHOTO @O210@
1 OBJE @O210@
1 OBJE @O223@
1 OBJE @O90@

[Nick-V]

Mike, due to an error I have edited my previous post (link below) about additional functions for the export plugin. CAUS of death does not need promoting whereas CAUS of DIV should be dealt with as mentioned. I now understand that FTM exports 1 CAUS but imports 2 CAUS and rejects 1 CAUS at input ("go figure" as they say).

http://www.fhug.org.uk/forum/viewtopic. ... 388#p59930

[Nick-V]

This posting may be edited as we learn more - I might well be wrong, please suggest edits.

To make a potential round trip of data from FH to FTM/Ancestry and back to FTM we need to simplify data using the simplest common approach and bear in mind that we have little control over FTM and what it might do in future.

At present simplified (my already simple) data in FH as follows:

1) Check and delete from the GEDCOM for any SENTence records created (accidentally).

2) Move all Name Suffices NSFX data to the NAME field like Fred /Bloggs/ Jr as FTM appears to create an alternative name for NSFX records found.

3) Append REPOsitory web addresses into the REPO email field (Sadly, REPO text gets lost on import to FTM). To see repository data in FTM sort Sources by Repository then double click on one (not easy to find this!).

4) I don't want marriage status in a note but do want ALL divorces as events. Create Divorce events for everyone with a _STATUS of Divorced (yes these two fields can differ) by displaying both fields in Record List - Individuals. I am not bothered about Unmarried couple and Separated so deleted these (not sure if notes records for FAM events import anyway).

5) Include the date in the title of all media as Media dates do not import to FTM.

6) and finally and perhaps quite radically, delete all Census events and related sources and images (back it all up to save regrets). Ancestry has a great facility to review hints and then accept data against multiple people in the case of a census. Unlike FH it does not create a Census record - which is not an event anyway, but creates a SOUR record with links (citations) against the individuals' facts. Consequently, perhaps all census sources and facts and images (painstakingly put in using the Ancestral Facts plugin) should be deleted before uploading to FTM. If you wish to start from scratch using Ancestry's arguably better approach then delete all Census events using plugin "Change any fact tag" then delete all census sources by moving to list and deleting all list records then deleted all media with 0 (using list again) then deleted all unlinked media using plugin "Check for unlinked media".

After exporting using the FH plugin and before import to FTM:

1) Rename "2 NICK" to "1 ALIA" so nicknames can appear in the index of names (marked as preferred).

2) Rename "1 _EMAIL" to "1 EMAIL" perhaps FH should do use EMAIL anyway.

[Nick-V]

This posting may be edited as we learn more - I might well be wrong, please suggest edits.

With my data (yours may be different) I wanted a simple and controlled output from the FH Export Plugin. I currently use it as follows:

Tab 1
GEDCOM FTM 2014
CHAR UTF8
Media Both
Folder c:\...\Family Tree Maker\{treename} Media

Tab 2
Named List (1 _LIST): Remove, Lists not supported in FTM
Custom Attribute (1 _ATTR): Custom event, i.e. Contact Details
Witness Roll (2 _SHA%u): Remove, Not used by me
To/From Place (2 _PLAC): Fact Note, Could move to Value field on events (not attributes) - if I understand the difference correctly
Fact EMail (2 _EMAIL): Remove, Not used by me
Fact Website (2 _WEB): Remove, Not used by me
Marriage Status (1 _STAT): Remove, I cannot see record notes imported against FAM. Marriage and Divorce events work. Unmarried, separated and never married statii are not so important anyway.
Date Field (%d DATE): Upper, However FH and FTM display BEF and AFT, they are always upper case in Gedcoms. Note that FTM sorts a year as if it were 1 January therefore a burial in 1900 can come before a death on 2 Jan 1900.
Media Date (%d _DATE): Remove, FTM only exports file and titl. Will not import media _DATE or DATE or other info.
Updated (1 CHAN): Remove, FTM does not support timestamps
File Root (1 _ROOT): Remove, FTM does not import or export home person - could be first in file?
Record Flags (1 _FLAGS): Remove, FTM does not support flags (option to make a custom event or set any security flags somehow?)
Place Record (0 @P%d+@ PLAC): Remove, Creating SOUR records is messy and benefits uncertain
Given Name (2 _USED): Remove, Not used by me
Record EMail (1 _EMAIL): Keep, then rename to "1 EMAIL" manually Need option 1 EMAIL for SOUR records
Record Website (1 _WEB): Remove, Don't think FTM uses WWW or WEB fields (v5.5.1?)
Source Type (1 _TYPE): Remove, There is no Type field in FTM although FTM's Category is not exported. Type of Source (family book etc.) not important.
Picture Note (%d _NOTE): Remove, Not used by me, did not seem to import
Media keywords (%d _KEYS): Remove, Not used by me, did not seem to import

Media and others fields could be handled in an improved manner. Some of this is discussed at: http://www.fhug.org.uk/forum/viewtopic. ... 794#p59930

[Nick-V]

This posting may be edited (and become two posts even) as we learn more - I might well be wrong, please suggest edits.

After importing the first time to FTM:

1) Match place names against their places database. This will replace the multilevel approach you used in FH which required the use of leading commas where data (such as town) was not known. My view was always to keep it simple (smallest place, county, country). Once normalised in FTM you get hierarchical access to places i.e. England, Surrey, Wimbledon although they are still stored and presented least significant first.

2) Set up security on the whole database and on private facts, images etc

3) Set one image as the profile picture for each person with media. Perhaps stupidly, FTM does not set a default to the first image for a person. Whilst other investigations are going quite slowly but well, I am currently "desperate" for a _PHOTO tag to be created in the FH export plugin (and links to the full images).

When matching and updating your data using FTM Ancestry:

1) Use Ancestry not FTM. Ancestry helps says "Records you’ve found on Ancestry and merged into Family Tree Maker won’t be re-uploaded to Ancestry.". Do matching in Ancestry. When doing it in FTM it appeared to create duplicate events !

[tatewise]

This is clearly developing into an extensive guide to exchanging data with Ancestry/FTM.

I would like to make it appeal to many users, and thus options based on personal preferences should be avoided in favour of general solutions.

This Forum dialogue is not the best way to formulate such a guide, so I have created a Knowledge Base section where the guide should evolve. See how_to:index&#advanced_guide|> FH Advanced Guide that leads to how_to:exchanging_data_with_ancestry_ftm|> Exchanging Data with Ancestry & FTM. Nothing is permanent. Its position in the KB, its title, and its structure are adjustable, and may well need sub-sections for different aspects of the exchange process.

If not already done so, you will need to learn about editing, and sign up for editing privileges:
info:what_is_a_wiki|> What is a Wiki?
info:signing_up|> Signing Up For Editing
contributeyourknowledge:index|> Contribute Your Knowledge
For most details, it is no more complex than editing these Forum postings.

[Nick-V]

Thanks Mike...I didn't know there might be a better way. I'm not sure my unverified work-in-progress is yet ready for such a place but its a good idea.

I hope this stuff is useful to others, but importantly I am documenting mainly to invite collaboration and improvements to my initial ideas, mainly in the approach but also the tools (like the various useful plugins I now use).

A week or two ago I knew nothing about GEDCOMs. I've gone at investigating somewhat madly and in a day or two I really must get on with updating my data with hints - I've held back but my data is there now. I believe increasingly this whole integration thing can be solved but I can't cover every aspect on my own - I'd really appreciate the _PHOTO tag so I don't have to move this forward manually.

In an effort to simplify this thread, you've probably noticed my edits to refer to the key topics which I will edit rather than create more postings.

Regarding personal preferences, I can only comment on my limited use of FH's fields and can only recommend what would work best in my opinion. I hope others will speak up for themselves to find a general solution. Further, a general solution (a program) will inevitably have to provide options...people use FH differently and prefer different things.

I will soon post some early notes about importing back into FH. So far I think a full replacement import is very feasible and certainly preferable to a merge. There are some issues but I see nothing more complex than the FH to FTM direction (if one accepts some minor compromises in exchange for powerful matching and easy updating).

How likely is it that you might do some development work if I continue the research? Frankly, the best way I can contribute is to investigate further and encourage others.

[tatewise]

The KB is just as collaborative as these Forums, and it structures the information much better.

I find it difficult to extract what I need for the Export Gedcom File Plugin because it is scattered across several postings and is still being edited.
If the Gedcom Tag conversions for FTM were in one place it would be so much easier. Something similar to glossary:gedcom_extension_list|> GEDCOM Extension List and plugins:help:export_gedcom_file:output_formats|> Export Gedcom File ~ Output Formats. In fact this second page, under Family Tree Maker 2014, is where a summary for that Plugin option will reside.

The other techniques to assist with the FTM to FH round trip would form a separate section in the KB, and include your advice on what features to avoid, importing methods, exporting methods, etc.

[Nick-V]

OK sounds good then, I'll take a look at the KB and perhaps paste some stuff across - I'm deeply buried in taking an initial look at getting the info back to FH today!

If you are happy for the KB to be a non-authoritative chat that's OK, I regarded it more as the final say...I just don't want anyone to take my unverified rapid research approach as gospel for their data or indeed any programming.

1) I believe the current info for image processing changes is at: http://www.fhug.org.uk/forum/viewtopic. ... =25#p59950

2) The current info for tag processing changes is at: http://www.fhug.org.uk/forum/viewtopic. ... 794#p59930

If there is anything else that has not been included in these two posts then it's tiredness! Any concerns or discussion required from your part please say....happy to retest. I have proved much with Notepad and Search/Replace.

Personally I would love a quick and nasty way to create those _PHOTO tags...I can't image how without coding...

Those two links to documentation look very good...wish I had found them earlier.

[tatewise]

Those pages are in the Help & Advice for the Export Gedcom File Plugin, but nobody reads the Help so you are not alone.

[Nick-V]

A few really rough notes about opening an FTM exported GEDCOM in FH:

1) FTM exports notes as note records which clutters FH and sometimes prevents access to the note data. Examples include RESI note into which the FH export plugin previously put the address. ALso INDI, NOTE and lots of empty notes too. These note records output by FTM need to be a fact or record notes. More investigation required on what else this affects.

2) TITL is used for surname, source title and object title. Selective promotion and/or tag renaming is required depending on parent record. More investigation.

3) CONC records appear where attribute data is too long and these appear to reject or the data gets lost.

4) FTM allows both attributes and events to have values (description field), it uses them (e.g. RESI to record Census info) and exports them all. Of course FH doesn't like such values against an event so moves them to the fact note and reports warnings.

5) Nicknames appears as 1 ALIA records. These can't simply be renamed to 2 NICK because they do not appear after 1 NAME - I've seen them after 1 SEX so they would have to be repositioned.

6) Citation media (Census) is not visible. In FH media is normally connected to a source but in FTM to a citation. IN FH what does the (greyed out) SHOW MEDIA button on the citation panel do? Data is:
0 @I19@ INDI
1 NAME Matilda /Ridgway/
2 SOUR @S10@
3 PAGE Class: RG14; Piece: 10332; Schedule Number: 36
3 OBJE @M2547@

7) REPO, EMAIL needs to be renamed to _EMAIL.

Sure this isn't 100% clear but I'd invite feedback about feasibility of automation before specifying properly.

[tatewise]

1) FTM note records ~ Clean Up Notes Plugin may help.

2) TITL is used for surname, source title and object title...
GEDCOM often defines the same tag in different contexts. Source & Object/Media TITL is perfectly OK. Individual TITL for surname is wrong.

3) CONC records appear where attribute data is too long...
Many programs misuse the CONCatenation & CONTinuation tags for long text, but they are only valid for specific Tags that support long multi-line text such as NOTE, ADDRess, Text from Source, and a few others.

4) FTM allows both attributes and events to have values...
Many programs make this mistake.

5) Nicknames appears as 1 ALIA...
Total misuse as ALIAs tag should link to another INDIvidual record.

6) Citation media (Census) are visible, but only via All tab and in Reports with correct options.
The (greyed out) SHOW MEDIA button on the citation panel opens Media on Media tab of Source Record.

7) REPO, EMAIL needs to be renamed to _EMAIL. Agreed.

[Nick-V]

Thanks Mike. In summary what I hear is FTM gets it wrong, and these things will have to be programmed around (if we are to succeed).

The two that bother me the most are:

1) changing note records back to record notes - yes the plugin solves it although I now believe that unconnected notes records (i.e. my to do list) don't get to FTM - certainly they are not exported from it anyway. More fun for the list.

6) FTM only creates 1 source record for the whole of 1911 - if we can only choose between INDI media and SOUR media (and not the citation) it's going to be tough...we can fix stuff but what about when it goes back to Ancestry again (effect on matches and existing online data!??!?). Thinking required.

[tatewise]

Yes, most can be suitably converted, but it looks like there may be some show-stoppers looming!

1) Try linking your uncoonected Note Records to a dummy Individual Record.

6) I am not a user of FTM, but that seems odd. Presumably it lets you create individual Sources for BMD events, so why not individual Census Households. FH does support Citation Media, but that Method 2 Census Source may be a show-stopper as far as round trip matching goes.

[Nick-V]

6) One fantastic thing in Ancestry is vital record matching and auto updates. With Census matching,you check its the right family, then it lists all the people and all the fields for each alongside your existing people and fields and you tick which you wish to update. On completion it creates citations from all census fields to all data for all individuals, stores the image and puts citation notes. Really quick and comprehensive.

So, I can set up individual sources...but this automatic feature regards 1911 census as one source of many citations.

Again, this is my current belief based on updating families with 1911 census info. I got 1 source, multiple citations and 2 images linked appropriately to those citations for the two families. I suppose if we cannot find a way for FH to record what it gets and re-output one option is to store such detail externally on import and merge back in on output. Shame!

[tatewise]

I suspect that FH can work with that Census Source structure, but it is not ideal.
Not only is the Census Media Image linked several times (once per Citation), but where is Household specific data stored, such as a transcript? It either has to be repeated in every Citation - Text From Source, or in a Note Record linked to every Citation - Note. Possible, but messy!