* Ancestry/FTM - FH -- A Different Approach

[shoshk]

Hi All,

I've been wading through the thread and knowledge base articles regarding Ancestry/FTM - FH Workflow and doing a lot of thinking and have decided to implement a different approach. This in no way implies a criticism of Nick's approach. I just have a different way of working.

I'm posting here because I will be doing a fair amount of custom programming (one or more external applications/utilities written in vb.net). I'd like to get input as I work through this process so that the final product may actually be of use to somebody else (in which case I would be happy to share). Of course, if nobody else is interested, I'll just go on my merry way.

What follows is a (very) high-level design for what I have in mind. I've started getting into a lot of the details (in my own mind at least) but haven't included that here so that people can get an idea of the basic approach and see if it's of interest to them.

A Little Background

I use Ancestry/FTM for much of my online research. Say what you like about ancestry, in my opinion, they offer an excellent search facility. Couple that with the ability to attach "finds" to my tree (my tree -- controlled by me) which can then be synced to the desktop via FTM (which takes care of downloading associated images) and it becomes the essential tool in my toolbox.

After using The Master Genealogist for many years, I have decided to move to FH. I looked at several other options, but kept coming back to FH and there I will stay.

I have decided to use TNG (The Next Generation of Genealogy Site Building) to publish my data. I'm just getting started here.

High-Level Design

1. Updating and adding new data will be done via Ancestry/FTM.
2. Data will be synced from FTM to FH (via an application which I am starting to develop)
3. FH will be used to add and update only data which is either not available in FTM (like named lists or flags) or which does not export from FTM (like media descriptions)
4. Finally, when I'm ready to publish, a GEDCOM will be generated from FH for uploading to TNG. How I handle certain items, like media files, will be affected by the requirement that things are setup in order to be compatible with TNG requirements.

Dataflow will be in one direction only, as follows: Ancestry/FTM --> FH --> TNG

The application(s) which will merge Ancestry/FTM data into FH -- at the core of all this -- will replace existing data for fields like DATE and PLACE, while leaving FH-specific data (like Flags) intact. The FH merge/compare file facility will not be used.

So, that's it. If this is of interest to anybody, let me know. I welcome your input.

Regards,
Shosh

[tatewise]

Hi Shosh,

I am a little saddened that you do not wish to adopt Nick's approach, because I suspect that is the way most users will proceed. It is possible that the Input Gedcom From FTM Plugin (still to be developed) will be useful to you. However, I understand that it is your choice.

I assume you appreciate the differences in the way Ancestry/FTM and FH handle Source Citations.
Ancestry/FTM focus on Citations with Media attached, which FH does not support so well.
FH prefers to attach Media to the Source Record.

Regarding, exporting FH to TNG have you investigated the Export Gedcom File Plugin options for TNG that have recently been enhanced?

[shoshk]

Mike,

Nick's approach assumes that the user will be moving data from FH to FTM.

I did a lot of thinking about this and realized that there is a problem with this approach. (But perhaps Nick has come up with a solution)

If the user updates an individual's birth date, for example, in FH, as far as I know, there is no way to get this information back into FTM via GEDCOM. The birth date will need to be manually updated in FTM. The same holds true for places, and possibly other fields. I don't want to have to make updates in two places. To keep it simple, I made the decision to make FTM the "database of record." I will use FH to add additional information (like flags, named lists, descriptions for media, place information (notes, geocoding), etc).

Nick's approach assumes that data is being exported from FH (including named lists, etc) and that it will be available to the import plugin. Perhaps I have misunderstood his approach, but it seems that he plans to completely replace the FH content on import. This won't work for me since I don't plan on moving data from FH to FTM.

Have I missed something here?

Shosh

[shoshk]

Mike,

Regarding TNG... I'm really just starting to look at this and there are only so many hours in the day. From the little that I have done, it is clear to me that the GEDCOM export module will prove very useful when I get to this stage.

Shosh

[shoshk]

Regarding citations...

I'm still thinking about this, but I'm definitely a splitter, so I will probably implement a conversion from FTM's "lumped" sources to "split" sources, so that I can attach the media links to the source.

Shosh

[jimlad68]

Shosh,

very interesting.

If it is not a daft question, would it not be easier to go straight from FTM to TNG, or does FH offer more features for other things, and is easier to move to TNG? For me it seems like a sledgehammer to crack a nut, but then I don't have your programming skills and I would imagine it could be fun if you are so inclined!

My only worry about this venture and the other two way sync project is that the products you are syncing with can change, I don't believe Ancestry/ FTM offer an API and regularly (well, certainly every few years) seem to have major database changes which may need regular updates to your own programs. I think your "Dataflow will be in one direction only" is safer and I should imagine a lot easier to control than a 2 way sync.

My principle at present is to keep 1 master database on FH. The only way I could imagine using your version would be to do research on Ancestry, capture into FTM and export to a new GEDCOM for "merging" with my master FH database. Even so, it would be nicer to export direct from ancestry (no need to purchase FTM), but I think images can be an issue.

Best of luck

[shoshk]

Jim,

On the face of it, it would certainly be much less work to go directly from FTM to TNG. However, FH offers additional valuable data options as well as much stronger reporting facilities, so the additional work is well worth it.

It would be wonderful if Ancestry published their API. We all know that an API exists and there are even a few products which have been developed using the API. I'm not sure what one needs to do to get access to the API. Perhaps one needs to be directly related to the lead programmer for FTM.

Anyway, when one develops an interface between two products, there are almost always changes that need to be made when there are new releases. That's the downside, but there doesn't seem to be much choice.

Over a number of years, I've collected a massive number of new records in Ancestry that I wanted to add to my desktop database (then TMG, now FH). I have added some, but most have been waiting for me to have the time. It's a huge, tedious task to undertake manually.

We would all like to export directly from Ancestry, but that's not an option.

I would much rather spend my time doing research rather than programming, but I figure that the effort I put into this now will pay off in the long run. At least I hope so.

Shosh

[tatewise]

Shosh, Nick would be better to answer how to manage the sync of FH and FTM databases, but he gives some clues.
See the first paragraph of how_to:exchange_with_ftm:step_3_import_your_fh_gedcom_to_ftm|> Step 3 Import your FH GEDCOM to FTM which suggests he uses FTM Merge to update FTM with any new FH data and preserve the FTM/Ancestry 'hints'.

Jim, using FH Merge/Compare to sync with FTM is an option that Nick has investigated and abandoned, because the two GEDCOM dialects are so different that the effort needed is excessive. Whereas, the Export Gedcom File Plugin formats the GEDCOM file to match the FTM dialect, making the FTM Merge much more feasible.

Shosh, in a similar way I suspect trying to merge the FTM GEDCOM with the FH GEDCOM will need some unique reference points. Have you considered how you will identify which record is which. For instance how will the merge know which Individual record needs updated DATE and PLAC fields, or which Media records have already been converted from lumped to splitter?

Nick, or rather the Export Gedcom File Plugin, resolves this problem by including the FH Rec Id in the NOTE of each exported record, so when the FTM GEDCOM is returned to FH the Import Gedcom From FTM Plugin will know explicitly how to reconstruct the replacement FH GEDCOM and all the links between records.

Shosh, the conversion from lumped (Method 2) to splitter (Method 1) is something that the proposed Import Gedcom From FTM Plugin might help you with.

I suspect the Ancestry API is a private proprietary interface for FTM only.

[shoshk]

Mike,

In fact I am working on the "unique id" issue at the moment.

All of this is complicated by the fact that I currently have 2 versions of my database -- FH (which was migrated from TMG) and FTM/Ancestry. Both contain unique data and one of my current tasks is to make sure that they both contain the same information. This task has needed doing for a number of years now and once it's done, hopefully, the FH-FTM sync will keep things, well, synced.

I have decided to add a new tag -- _linkid -- which will contain the unique identifier to be used for matching records. I want to use a custom event which will only be used for that purpose. I am afraid to use REFN, which could potentially be changed by another process.

If all my data was(were?) just in FH, I would simply write a plugin to generate _linkid, which would be transferred to FTM via the intial GEDCOM export.

Since that's not the case, it's a bit more complicated, but I'll get it done.

On an ongoing basis, when new records are added to FTM, the user will need to manually add a _linkid tag to each new individual. At least that's how I see it at the moment. Perhaps I can come up with a way to automate this process. But, even if not, I don't think it will be too bad.

_linkid will be attached to each individual in each of the three environments -- FH, FTM and Ancestry. This is important because I have realized that the Ancestry GEDCOM contains some very interesting information not necessarily present in the FTM GEDCOM (the information necessary for building a URL to access records associated with source citations). I will probably not export most of my downloaded media to TNG (copywrite issues) and so providing a URL that will enable people with an Ancestry subscription an easy way to access records that I have found is a distinct advantage.

In fact, my merge process will take as input both a GEDCOM from FTM and a GEDCOM from Ancestry, merge them and then merge the result with FH.

I will be taking the approach of overwriting existing FH data (dates, places, etc) so that will not be an issue. I simply need to isolate the elements that are unique to FH so that they can be preserved. That's why I've presented what I'm doing on the forum. It will be relatively easy to tailor this process to the way I work and the features that I use in FH. Unless it's a lot of extra work for me, I'd like to get input from other people who may be interested in using the final merge application so that it could serve their needs as well.

BTW, the Ancestry API is certainly proprietary, but it is definitely being used by products other than FTM. Clearly, Ancestry does not want to release it to the general developer population, even under non-disclosure agreements, but there are a few lucky souls who have managed to get their hands on it.

Shosh

[tatewise]

Shosh, Individual Records are only one of up to 10 different GEDCOM record types:
Family, Note, Source, Repository, Media, Place, Submitter, Submission, Header.
The first 6 are critical, but the last 4 may be insignificant.

For example MARRiage events are held in Family Records that will need unique identification to update DATE & PLAC.

Those first 6 GEDCOM record types do support REFN Custom Id tags, but they certainly do NOT all support Custom Events.
I have serious doubts about whether FTM and Ancestry support REFN or a user defined tag such as _LINKID that can be added to at least the first 6 critical record types.

Every new record will need a unique id, not just Individuals.

[shoshk]

Mike,

I believe that records may be divided into the following groups vis-a-vis treatment by the merge routine.

1. Records which do not need to be processed -- leave them as they are
- HEAD
- SUBN
- SUBM
- PLAC

2. Records which will be replaced in their entirety

- SOUR
- REPO
- NOTE?

3. Records with tags that are being maintained in FH which we want to preserve, eg. INDI flags

- INDI
- OBJE
- FAM?

I am not using any FH-specific tags for FAM records, so my current inclination is to put them in the second group (overwrite).

Still working out the precise details, but that's more or less where I am at the moment.

Shosh

[tatewise]

1. Yes, you can probably get away with that, but don't quote me.

2. OK. SOUR will need their OBJE links translated. FTM/Ancestry have a nasty habit of turning local NOTE fields into NOTE records, which may need to be undone.

3. Yes, INDI, FAM & OBJE need unique id and plenty of processing.

Some overriding considerations...

Earlier you mentioned preserving FH Named Lists. These can contain any combination of all 10 record types, especially useful will be INDI, FAM, OBJE & SOUR.

Diagrams cross-refer to records by Rec Id, so I think these must be preserved for INDI & FAM (needs confirmation).

It sounds like you do not plan to use FH Face/Detail Frames for OBJE Media, because the cross-reference _ASID are used in SOUR, INDI, FAM & _PLAC records.

Are you interested in the Preference order of OBJE Media in the Media tab of SOUR records? If so, that is another feature that needs preserving.

I still doubt if FTM and Ancestry support REFN or a user defined tag such as _LINKID in the required records.

So users will need to add unique id not only to each new Individual but also each new Media, and I suspect each new Family and may be Source.

Please do not take these comments as negative, but as pointers to potential technical complications you may have overlooked.

[jimlad68]

Shosh,

Just a thought. One of the things I have always done is to organise my data such that it performs as easily as possible the things I want to do with it, be it portability, reporting, diagrams etc. For instance, I wanted to add text to sources I created for PLACes, perfectly standard Gedcom, but even FH did not display them in reports (if anyone knows of a way to do it I would be grateful), however, although it exports it did not work in other programs I tested. In consequence my FH fix, (and coincidentally cross-platform), is to attach the source to relevant facts instead, hence it then shows in reports.

So, where I am getting to with is: If you can limit the use to "features" that are easier to sync/migrate, you could save a lot of trouble.

[shoshk]

Jim,

This is definitely going to be my approach, at least initially. Since I've just started using FH I have the opportunity of deciding what features to use (and how) in order to keep the syncing process as simple as possible.

Shosh

[shoshk]

Mike,

Using Named Lists

At the moment I'm using Named Lists for individuals. I hadn't thought about the implications (or potential advantages) of using them for other kinds of records. I understand if I decide to use named lists for any kind of record, I will need to preserve the FH record id. So, there is a cost/benefit relationship here.

I guess that I should be trying to preserve the FH record ids as much as possible. As long as I have a way to map the FTM XREF to the FH XREF, I can do that.

Mapping XREFs

Ideally, this can be done by using existing information, without requiring user intervention (ie. defining a _linkid).

OBJE records -- I haven't thought this through completely yet, but I believe that <filename> (full path name) should uniquely identify media records.

SOUR records -- The best thing I can think of is to use the source title. Ancestry sets this up automatically for records it adds. I believe (hope) that they are unique. Although the title does not have to be unique (there is nothing to enforce this in the system), I can take care of this in the merge application by making sure there are no duplicate source titles and refuse to continue until the problem has been corrected. I will probably also provide a utility (say, "Check for Duplicate Source Titles") to support this process. If, somewhere down the line, I want to change a title, then this will need to be done in both databases, but I don't think this will prove to be too much of a burden because, at least in my experience, source titles tend to be quite stable over time.

INDI records -- I have been trying to figure out if there is a way that I can uniquely identify individuals without defining a new id for them. So far, I haven't come up with anything. In most cases, the combination of Name-Birthdate-Deathdate will uniquely identify a person, but not always. I have cases, for example, where I have identified 2 cousins, both named William, both named (lets say) Smith, born the same (estimated year) and consequently, both with the same, estimated deathdate (I use "before <birthyear>+100"). Thinking out loud, perhaps the approach could be to define _linkid only for individuals who cannot be uniquely identified by the Name-Birthdate-Deathdate combination. Of course, this breaks if Name or Birthdate or Deathdate changes. Sigh... I need to think about this some more.

FAM records -- This is tough. As with INDI records, I would prefer not to have to define "unique ids" for families. I want to automate the process as much as possible but it's not at all clear what could serve to uniquely identify a family, in lieu of a single unique id. Family composition could very well change over time as new records are found. So, I'm starting to wonder. Do I really need to preserve FAM:XREFs? I've yet to find a reason to use them in named lists. Is there any other kind of data I might want to preserve for families? Oh yes, you mentioned Diagrams. I haven't played with diagrams very much yet. This is definitely an area for research. I guess I'm going to have to learn about diagrams sooner rather than later.

Other Points

Face/Detail Frames -- Need to learn more about this; not using it at this time.

Preference order... -- Have to think about it.

_LINKID -- FTM supports adding a custom tag for both INDI and FAM records. (I just don't want to) I think I've got SOUR and OBJE records covered and won't need a custom tag.

Mike, believe me when I say that I truly appreciate the effort you put into commenting about my ideas. I've been doing this for a long time (software development) and know that the best results come from collaboration with others and the questions that are raised. You have definitely raised points that I haven't considered. Thank you!

Shosh

[tatewise]

Shosh,

Named Lists

Remember that Facts can either be in Individual records or Family records, so if you need a list of say MARRiage or DIVorce facts to follow up, then the Named List will comprise Family records, or if as a 'splitter' the details are in Source records, then the list will comprise Source records.

OBJE Records

There is nothing in GEDCOM that demands only one OBJE record links to one Media file. This would have to be a manual discipline, perhaps with a check utility. Note that the FH GEDCOM can use full absolute path names, but they complicate Projects migrating from PC to PC, etc, so FH recommends relative path names with the Project Media folder as the root. This will need a bit more XREF mapping.

SOUR Records

These actually have two titles (TITL and ABBR) but FTM/Ancestry only use TITL so you should be OK.

INDI & FAM Records

Yes, the only unique thing about these is some unique id as everything else can change or be blank. Several features refer to both INDI & FAM record id. It is Diagrams saved as Charts that use record id so that a formatted Diagram can be saved, but will auto-update with new details from the linked records. Named Lists as mentioned above.

Other points

As I am sure you aware, all features need to be considered to ensure they do not become 'show-stoppers'.
However, to invite collaborative partners may need covering more bases, because they may not be happy with your constraints (as I know from my published Plugins).

The fundamental technique used by Nick (and my Plugin) is to preserve the FH Record Id (and most other details) within text in a NOTE field for each record. As the challenge of your approach develops, do not lose sight of Nick's alternative approach.

[shoshk]

Mike,

Preserving information in note fields

What happens if I'm using note fields for "real" information?

If I understand correctly (from wading through the workflow thread) you are appending the information you want to retain to the end of any existing notes. So, my stuff will appear at the beginning of the note. But, if I want to print out my notes, all the "synthetic" stuff will be printed, too. It seems like it could get rather messy.

If you and Nick have discussed this issue, please point me to the appropriate page in the thread (it's gotten so long that it's difficult to find specific issues). Thanks.

FTM Citations

Well, I found out (had suspected but just confirmed it today) that FTM handles source citations very differently from what is generally accepted.

In fact, it is easy to support splitting at the source citation level. I can link the same citation to multiple individuals and facts. In a way, it's rather nice. I can have a much shorter source list because the splitting is taken care of at the source citation level.

On export, FTM generates multiple citations which therefore arrive in the destination "lumped."

On import, it appears that FTM compares Source + Source Citation everywhere they're used and if they are the same, it puts them back together (internally), so everybody and their facts are linked to the same citation. I removed the CONC lines from the PAGE tag for several citations and it put those particular citations together. So it looks like the comparison is done on the raw GEDCOM text.

Anyway, the implication here is that if source citations are converted to sources for import to FH (lumped to split), the reverse conversion (source to source citation) needs to be done when sending them back to FTM.

Shosh

[tatewise]

Sorry Shosh, I assumed you had been getting your information from how_to:exchanging_data_with_ancestry_ftm|> Exchanging Data with Ancestry and FTM which gives a clearer explanation than the Forum postings. See also plugins:help:export_gedcom_file:ftm_family_tree_maker_2014|> Export Gedcom File ~ (FTM) Family Tree Maker 2014 for details of the FTM Gedcom formats.

I suspect Nick is not so concerned about the Notes format in Ancestry/FTM because his master data is in FH, which does not have the 'Name Tags' added to Notes.
However, the 'Name Tags' have a very specific format, and perhaps could be filtered out of reports. Further formatting could be easily added if there is a mechanism in Ancestry/FTM to exclude selected details. The equivalent in FH is [[double square brackets]] for private details to optionally exclude from reports.

Yes, the Plugins are intended to handle that lumped/split conversion in both directions.
For example, the Export Gedcom File Plugin copies Source record Media to the associated Citations.

BTW: I don't think you can link Citations to Individual or Family records in FTM as it only support Citations on Facts.

[shoshk]

Mike,

You are correct, citations are linked to facts (either individual or shared [fam]).

FTM reporting facilities are fairly limited when it comes to customization. I believe that the only option you have on reports is whether or not to include them. You have the option of marking an entire note private, however, I have not found an equivalent for [[---]] for marking sections private. If the capability exists, I'd be happy to use it.

I understand that this is probably the best solution if you want to be able to transfer data in both directions.

I would really prefer to keep the notes for their intended purpose, so I have to think about this a little bit.

Shosh

[tatewise]

From searching Google it seems that FTM has Person, Fact, Relationship(Family), and Research Notes, and any Note can be marked Private. It also has Source and Media Notes.

Which of those Notes are you most concerned about?

I wonder if there is any way that Research Notes can be employed somehow, but maybe they are not available via GEDCOM?

In practice I suspect the 'Name Tags' may not be too intrusive, and may carry useful details existing in FH that cannot be represented in FTM. Why not try some experiments with a sample Project, perhaps the Family Historian Sample Project?

[shoshk]

Mike,

I'm still feeling my way around FTM and learning what it can do. I'm figuring out how to accomodate both my and my husband's way of working (which are quite different).

INDI records

Without thinking about it too much, I would say that I will probably want to use both kinds of person notes. Using a name tag to pass information back and forth seems like a good solution here. In most reports, only the preferred name is displayed, so I don't think it should pose too much of a problem. It would certainly be less obtrusive than using the person notes. I think that a custom EVEN could work here as well. That might be even better.

FAM records

Although I haven't been using FAM notes up to now, I know that my husband does and he would absolute hate it if he had to contend with anything else in the field. He would also definitely find a way to mess it up on a regular basis. Although sometimes frustrating, it's good to have a real USER in the house -- keeps me honest.

Anyway, I believe that it would be better to keep the FAM note free for its intended purpose. What about using a custom EVEN to pass information? It's easy to exclude facts from the family group sheet.

SOUR records

Interestingly enough, it seems that there is currently no way to include source notes in reports, so we are probably safe using a note to pass source information back and forth. It's one of those FTM quirks -- provide a place to store data that you have no way of displaying.

OBJE records

FTM does not export the Description field for media records. It does, however, export the note and I was thinking that I would use that for my media descriptions (and convert it to a description on import to FH). I will very likely also want to use the note field to record the source of the image. I think that we could probably figure out a way to co-exist. It's just a matter of defining a convention for the format of the note.

Shosh

[tatewise]

I think you may have misunderstood the concept of 'Name Tags'.
They are stylised Note text to represent FH GEDCOM data that needs to survive the FH-FTM-FH round trip.
See plugins:help:export_gedcom_file:ftm_family_tree_maker_2014|> Export Gedcom File ~ (FTM) Family Tree Maker 2014 for details of the FTM Gedcom formats.

A synthetic Custom Event for INDI/FAM record Notes would probably work, similar to the synthetic Custom Event for INDI/FAM record Citations exported from FH as FTM does not support record Citations.

You did not mention your use of Fact Notes.

I believe there are also NAME local Notes that have not been mentioned so far.

May be I should leave you to experiment for a while until the FTM/Ancestry and FH features are better understood.

[shoshk]

Mike,

I guess I didn't mention fact notes because it just seemed obvious that I would be using them for the standard purpose.

Yes, notes can be attached to the Name. I guess that's what I thought you meant when you were talking about Name Tags since that's how I refer to them when talking about the GEDCOM spec. OK, that didn't come out very coherently.

I like the idea of using synthetic custom events for transferring INDI and FAM record information. It seems to be the most unobtrusive.

I will re-read the knowledge base article.

Shosh

[shoshk]

Mike,

I have a clear direction for further research and investigation now. I have a number of other things to do, but will start chipping away at this task.

Do you intend to preserve converted FTM structures (say source citations) so that they may be restored during export?

Shosh

[tatewise]

I do not use FTM/Ancestry in the way that Nick does, so am reliant on his feedback on many details, but I would imagine preserving FTM structures in the GEDCOM exported from FH will simplify the merge into FTM.