* Ancestry, FTM, FH and workflow ~ Data retention

[tatewise]

This thread continues from Ancestry, FTM, FH and workflow (12388) but focussed on Data retention issues associated with project data that can complete the round trip from FH to FTM back to FH (other than Media and Sources).

[tatewise]

In how_to:exchange_with_ftm:step_2_create_a_gedcom_using_the_fh_export_plugin|> Step 2 Create a GEDCOM using the FH Export Plugin, and elsewhere, it advises that many data fields should be Removed from the exported Project sent to FTM. The strategy also suggests that on return to FH, it replaces the original Project.

That cannot be an acceptable strategy for most users, just to obtain a few Ancestry/FTM hints!

There must be a way to retain as much as possible of the FH Project data structures.

In particular, the following are important points, but not in any precedence.

1) Named Lists are used by many users to customise Diagrams, Exports, Work in Progress, etc, and must be retained.

2) Marriage Status _STAT is important for the wording in Narrative Reports to avoid using married/husband/wife for Unmarried couple and Never married status.

3) Place Records _PLAC you say "benefits uncertain" but users will need to retain their Lat/Longitude, Notes, linked Media, etc.

4) All other data must survive the FH/FTM journey, including Updated CHAN date-time, Witnesses, Email & Web, etc...

[Nick-V]

I am using the KB documents for development purposes using my data, as a working record for me and for collaboration with you...not for advising anyone. I am saying what works for me and hope we can build on that. I want to start at the basics and work up or its simply too complex. We'll take it offline if it poses a risk.

I don't need the other fields and in an effort to simplify research I am starting with everything possible switched off. Once clear, we can take each option and see if it works or what it might take. It is likely some data needs keeping outside FTM (separately or in FH) and this is where a merge would come in, after all attempts to make the basics work without a merge. It's just my approach, I come with 2 weeks experience of GEDCOMs, need to make it easy and need to see progress.

Lists do not exist in FTM.

_STATUS does not exist in FTM. Custom "facts" can be setup and other ways may be found to preserve this. In FH I note it is easily possible for _STATUS to conflict with MARR and DIV records. Given that conflict I took the pragmatic step (at present) to scrap use of _STATUS.

_PLAC records. Again I use FH simply and stage one needs to achieve this for the potential masses of Ancestry users who want what FH offer. Yes retaining FH info not in FTM needs consideration (merge into FH, external data, etc.). For those not linking pics to their place records, recording text there or using the Map plugin or ancestry for lat/long there is no issue. Those using every corner of FH might have to wait longer !

I am aware there are many as yet unseen and unsolved issues. I think it would unwise and a disservice to the simplistic majority to risk slowing or indeed killing off a simple solution because we don't yet know how to make it work for the complex minority. I understand the concerns. My preferred approach is to get started and do as much as we can...

Others might share my position which I'll put simply:

1) I want to get my data into Ancestry as fully as possible even if I need to compromise some areas.
2) I want to use Ancestry to match thousands of hints and to record the data changes and citations with media quickly and consistently.
3) I want my data back in FH because it produces my web site, is more flexible and independent of any one provider (I can upload data and images elsewhere).
4) On-going (non bulk) hint processing may well be in FH manually.

[tatewise]

The public KB approach is, I believe, the right way forward, and may entice other users on board.

Perhaps the KB needs to make it clearer which advice is firm, and which an expediency to simplify the initial stages.

I am a retired software professional, and have had experiences where an initial over-simplification leads down a blind alley. I suspect that formatted NOTE text/records may offer a solution, and once ratified for a few items, can be rolled out across all the data that needs to survive the roundtrip, both FH custom data and FTM custom data.

We have already identified a number of essential items in other threads that need such a survival technique.

FYI: Place Lat/Long values come from within FH as well the Map plugin and elsewhere.

[Nick-V]

Yes it would be good to get help, I recognise how you and Jane may feel sometimes.

Yes, clearer, right now I suggest none of it is firm and anyone wanting to press forwards has the leading edge experience of a test pilot!

Yes, I suspect formatted note records are probably our only option other than external data. An option I recognise and would prefer to use as infrequently as possible as the data used by programs could be changed manually - and its usually unusual data that screws the program!

Yep FH, the map plugin and Ancestry all have their own approach to long/lat. I use the map plugin in a way that shares FH data (a good feature). Does the user need maps from FH/Plugin, from Ancestry (which aids hints) or both...version 2+ as far as I am concerned.

I respect your approach, appreciate your input and understand exactly why you seek full detail. In many situations the risks of getting it wrong and having to rework things can be less consequential than being pedantic. Prototyping is a great thing...and many projects can be forgiven for overrunning time and budget - had the truth been considered they would never have started. My gut says this is one of those areas we need to progress incrementally (we already know many significant issues, have outline solutions to most/all and surely we have the confidence to rework or just say tough luck...it exists, it does a lot, and it ain't perfect for everyone! I think we share the goal and have energy...just need to consider approach...

[tatewise]

Since you use the Map Life Facts plugin synchronised with FH Place Records, then I don't understand why you are proposing to discard them all.

[Nick-V]

As per KB, Ancestry keeps place data differently to FH (no level structure, no leading commas). Normalisation is done against their place database which probably helps with matches and almost certainly provided lng/lat data. I've decided to fit in the Ancestry and get rid of my three level leading comma town, county, country structure.

When normalised places get back to FH I can simply lookup the long/lat again and then both places work.

[tatewise]

Please post some examples of PLACe fields in Gedcom imported to FTM, and the same ones exported from FTM after Ancestry normalisation.

I must say I feel uncomfortable about the direction that is taking.

[Nick-V]

All it does is change your existing data (should you choose) to match against a database of names:

In:
Town, County, Country
, County, Country*
, , Country
Village Incorrect, County, Country

Out
Town, City, Country
City, Country
Country
Village Correct, Larger Place, Big Area, County, Country

So you loose leading commas and level structure. But you gain verification against known places database.

[Nick-V]

I have put data into as many FTM fields as I can find. Interestingly there are differences between an export (file 1 below) and a import/re-export (file 2), I note:

Source notes export but do not reimport
Some tags change a little

Export 1.txt

Export from FTM

(5.37 KiB) Downloaded 227 times

Export 2.txt

Export from FTM, import, re-export

(5.33 KiB) Downloaded 223 times

[tatewise]

If no bigger than 256 KBytes you can attach up to three per posting to this thread using Upload attachment below edit box. If it won't allow .ged then change to .txt or compress into a .zip.

[Nick-V]

Mike, I've looked into local NOTE and NOTE record usage for the round trip - see at the bottom of KB Step 6. In summary it looks like we can note data on the 0 INDI, 0 FAM, 1 Event, 2 Citation, 0 OBJE, 0 SOUR but not REPO or _PLAC.

[tatewise]

Nick, I may be a little dim this morning, but can't follow your round-trip thinking.
To use NOTE for round-trip data, we must first get NOTE data from FH to FTM, before getting it back.
0 INDI, 0 SOUR, and 1 <Fact> local NOTEs from FH to FTM seem OK.
But 0 FAM and 0 OBJE local NOTEs from FH you say don't import to FTM.
Not sure about others, except 0 _PLAC will never be used.
It seems everything comes back as linked NOTE records, without any local NOTEs at all.
May be you are suggesting the Plugin exports all round-trip data in linked NOTE records?

[Nick-V]

I see no signs of dim !

Please remember I am not documenting solutions. I am documenting at the earliest opportunity and for discussion my observations and enhancing these until they hopefully become solutions at some point later.

Yes clearly we need to put data in a note first, this being the preferred method of retaining FH data. But let's not do that until we know what notes are available and how they return to FH. This is a start.

Despite previous observations it does appear that 1 NOTE and 1 OBJE (but not 1 SOUR) can appear at FAM level as they can at INDI level. This is well hidden in FTM. There is a FAM screen that looks like an afterthought (poorly presented and bare) that one can access only from the menu (EDIT, EDIT RELATIONSHIP). This displays FAM events and their citations just like we see on the INDI screen anyway, but again, hidden at the very top left are two tabs for NOTES and MEDIA. This is good news.

At present I believe Citation notes and SOUR notes are local notes (documented) not NOTE records. There are unnecessary blank notes. Most note records need to be made into local notes again (the other plugin we discussed).

I am not suggesting anything, I am researching. In fact I suspect we'll use BOTH note records and local notes much as we currently do rather than make everything a note record.

Just trying to establish what we have to work with...none of it is obvious and much testing is necessary to discover things. I don't think any more precise information is available.

[AdrianBruce]

... But you gain verification against known places database. ...

I think many of us suspect that Ancestry's normalised place-names are no more than a list of place-names that Ancestry has scraped from some data-sources like censuses - and are therefore subject to indexers' errors, duplications and programmers' interpretations of an American genealogist's view of British family history.

Based on the hint list that comes up when I make a census enquiry, it's clear they've not scraped everything. People in my database have birthplaces of Monks Coppenhall, Cheshire and Church Coppenhall, Cheshire. These are genuine administrative units but Ancestry doesn't recognise them in a query place-name hint-list - instead it only has Coppenhall, Cheshire (the parish containing those two) - and Coppenhall Moss, Cheshire, which is basically a bog north of Coppenhall!

When I look at my tree as it's been loaded into Ancestry, the births in Church Coppenhall have come through unscathed and appear as Church Coppenhall, Cheshire, England. On the other hand, the births in Monks Coppenhall appear to have been normalised to Crewe! (I can't remember the exact sequence but the Monks Coppenhall Registration sub-district may have turned into the Crewe sub-district.).

[Nick-V]

Adrian

Thanks for letting us know about this - I noted anomalies too.

With the "place" aspect of what we are trying to do at least users have some choice. Either fit in with Ancestry however imperfect, or use the structured leading comma approach in FH. I'm undecided...do you have any advice...?

[Nick-V]

Mike, as mentioned, with round trip in mind, at Step 6 of the KB I have tried to identify NOTE and other interesting records that FTM handles. Alongside I mentioned observations about the way these are exported from FTM. I have just started looking at the way such FTM data is imported as this might differ knowing FTM (and guess what!).

We need notes for objects to record a variety of field including media date, picture note and media keywords (existing plugin options). We are also discussing others for the round trip including _DATE, _KEYS, _ASID, _AREA, _EXCL, _CAPT.

Like most of this project I have spent hours checking out small things and there are two issues:

1) FTM exports Note Records for each OBJE. On import, it appears to accept both Note Records and Record Notes but will accept neither unless 0 HEAD, 1 SOUR FTM (not 1 SOUR FAMILY_HISTORIAN). Conversely, the opposite value must be applied for Citation Reference notes to import as exported and not need "resetting" (see KB Step 6 FTM Issues).

2) The plugin option "Move to Note Record (GFT)" appears to export local notes to the OBJE - not note records. Affects media date, picture note and media keywords.

[tatewise]

1) Plugin could easily produce 1 SOUR FTM if that helps, and remove the subsidiary tags 2 VERS, 2 NAME, 2 CORP.

2) You have found a latent bug with GFT Note Records that should be fixed in next version.
[EDIT ~ Also GFT Note Records only apply to Local Media Objects as explained in the Help.]

[Nick-V]

Thanks Mike. As you can imagine the first issue was a pain to find and remains a pain to address.

I am concerned what else FTM "secretly" does differently with it's own data...some things may be useful, some an obstruction and certainly this is not consistent GEDCOM handling !

I am also without a solution on this at the moment...with "FTM" in that tag OBJE NOTES work and we need them, however, Citation Reference text gets read from NOTE or _FOOT rather than being "composed" from multiple fields (this can be "reset" manually on each record individually). I need to check the latter again...to emulate the "reset" situation it might be that we "compose" in the same manner.

[Nick-V]

The matter of the Citation Reference Note needs discussion before I clutter the KB with information and proposed solutions. I don't know all the facts but hopefully have sufficient to set out the issue and get assistance with a solution. The rough outline follows (corrections welcome):

Citations are links from a piece of your data to some proof, usually a document. For example, in FH you can link an event like a MARRiage to multiple SOURes each of which can be linked to a Repository. Additional information including a NOTE can be added to that event link to help describe the citation and how it is using a particular SOURce (there can be many citations to a single source). NOTEs and media OBJEcts (pictures) can be attached to the event and separately to the SOURce. When you select an event the citations displayed are named by the SOURce title or a short title.

FTM is similar but there are differences that need to be considered. Ancestry tends to employ one SOURce for England Census 1911 whereas in FH we tend to have one SOURce per document (family in 1911). This means citation identification and description is more important in FTM. Indeed some of the media OBJEcts attached to the SOURce are also attached directly to the event citation for this reason. When you select an event the citations display their objects and the name can be a complex compilation of citation, SOURce and REPOsitory data.

It is this important citation name, known as a "Reference Note" that is the issue. An example is:

Code: Select all

Jacob Jacobson, Die Judenburger Bucher der Stadt Berlin 1809-1851 (Pub Information), London Library (for Berlin Jews books etc.), 14 St James's Square, London SW1Y 4LG, Where in source. Text from source more lots etc. web address.

Technically this appears to be (the last two fields are user options):

Code: Select all

{SOUR, AUTH} & ", " {SOUR, TITL} & " (" & {SOUR, PUBL without headings} & "), " & {REPO, NAME} & ", " & {REPO, ADDR} & ", " & {INDI, Event, SOUR, PAGE} & ". " & {INDI, Event, DATA, TEXT} & ". " & {INDI, Event, SOUR, _LINK} & "."

When the GEDCOM is exported and reimported everything works fine. However, if a citation NOTE was entered this overwrites the "Reference Note", potentially rendering it meaningless. There is a reset button but of course we don't want to have to press that on every individual after an import! Further, the NOTE field is cleared as the note becomes the new Reference Note and entry of a new note results in a _FOOT record allowing two notes!

The import program uses a HEAD, SOUR tag to know if the GEDCOM was created by FTM or FAMILY_HISTORIAN. When FTM we see the unfortunate overwriting behaviour above, however, when something else it compiles the Reference Note freshly as we'd prefer. I suppose this design is deliberate, when importing the first time from elsewhere it compiles but when importing your existing FTM GEDCOM it does not interfere. The problem is that we HAVE to use FTM due to a separate problem with OBJE, NOTEs not being imported.

"On the upside", it appears that if we delete BOTH the NOTE and the _FOOT then FTM will reinstate the compiled Reference Note irrespective of the HEAD, SOUR tag. Clearly this renders the NOTE field redundant but other usable fields include citation detail, citation text and citation web address the latter two of which are optionally appended (no idea what happens after an import) to the Reference Note.

The data discussed above (if we delete NOTE and _FOOT appending them to TEXT) should survive a round trip:

Code: Select all

0 @I1@ INDI
1 NAME Fred /Bloggs/ Suffix
2 SOUR @S7@
3 PAGE Citation Detail
3 DATA
4 TEXT Citation Text
5 CONT ...more text
3 _FOOT First note became _FOOT
3 NOTE
4 CONC Second Note
4 CONT
3 QUAY 3
3 _LINK www.webaddress.com

This thinking has been a little complex - others are invited to check and comment...

[tatewise]

I cannot comment directly on FTM but this is my take on the scenario.
FH will never export _FOOT tags.
FH via my Plugin can inhibit Citation NOTE tags, that are not widely used, and when necessary put the NOTE text elsewhere to survive roundtrip. We just need to decide where.

I understand that the FTM Census Method 2 tends to differ from the more popular FH Census Method 1, and poses most challenges, but when presenting these concepts could you also address the citation of BMD and other Sources. i.e. Is it only a Census problem, or is it more widespread?

[Nick-V]

Thanks Mike.

I tend to refer to Census when thinking about citations because I regard them as more complicated than some other citations purely because one document affects several people. I don't wish to exclude BMD or any other citation in doing so.

As far as the document>citation>event relationship in the software is concerned I am not sufficiently knowledgeable to conceive any difference between a Census and a BMD or other event.

Please let me know if there are specific situations that I should consider.

[tatewise]

There is no difference in citation structures for different sources.

The important difference is whether Method 1 or Method 2 is used.
Clearly from your description FTM uses Method 2 for Census years, and so do some FH users, but adding Media and Transcripts per Household pose a greater challenge than Method 1. This is illustrated by the FTM technique of duplicate Media links, once in the Source, and once in the Citation.

Does FTM use Method 2 for BMD sources? i.e. one Source for all births in one Church Parish, or some other large collection.
Or does FTM use Method 1 for BMD sources? i.e. one Source for each single BMD Certificate/Church Record.

I agree that a Census source tends to be related to more people, but BMD are also related to multiple people/facts. That is not what complicates the FTM citations. It is the Method 2 where the Source relates to many more people than its Citations.

[Nick-V]

One thing I mention elsewhere once but not above is that FTM doesn't import Census events. Potentially there is a data loss here (attached notes, objects etc).

OK almost understood...it seems to group into separate databases. Here are some sources:

"Connecticut, Marriage Index, 1959-2012"
"England & Wales Christening Records, 1530-1906"
"England & Wales Marriages, 1538-1940"
"England & Wales, Birth Index, 1916-2005"
"England & Wales, Death Index, 1916-2007"
"England & Wales, FreeBMD Birth Index, 1837-1915"
"England & Wales, FreeBMD Death Index, 1837-1915"
"England & Wales, FreeBMD Marriage Index, 1837-1915"
"England & Wales, Marriage Index, 1916-2005"
"England & Wales, National Probate Calendar (Index of Wills and Administrations), 1858-1966"
"England & Wales, Non-Conformist and Non-Parochial Registers, 1567-1970"
"England, Select Births and Christenings, 1538-1975"
"England, Select Marriages, 1538–1973"
"Florida Passenger Lists, 1898-1964"
"Germany, Select Births and Baptisms, 1558-1898"
"Hamburg Passenger Lists, 1850-1934"
"Index to Alien Arrivals by Airplane at Miami, Florida, 1930-1942"
"JewishGen Online Worldwide Burial Registry (JOWBR)"
"Lancashire, England, Deaths and Burials, 1813-1986"
"London, England, Baptisms, Marriages and Burials, 1538-1812"
"London, England, Births and Baptisms, 1813-1906"
"London, England, Deaths and Burials, 1813-1980"
"London, England, Electoral Registers, 1832-1965"
"London, England, Marriages and Banns, 1754-1921"
"Louisiana, Marriages, 1718-1925"
"Louisiana, Naturalization Records,1836-2001"
"Louisiana, Statewide Death Index, 1900-1949"
"Mecklenburg-Schwerin, Germany, Census, 1867"
"Mecklenburg-Schwerin, Germany, Census, 1890"
"Mecklenburg-Schwerin, Germany, Census, 1900"
"New Orleans, Louisiana Birth Records Index, 1790-1899"
"New Orleans, Louisiana, Death Records Index, 1804-1949"
"New Orleans, Louisiana, Marriage Records Index, 1831-1920"
"New Orleans, Passenger Lists, 1813-1963"
"New York, Index to Petitions for Naturalization filed in New York City, 1792-1989"
"New York, Naturalization Records, 1882-1944"
"New York, New York, Death Index, 1862-1948"
"New York, Passenger Lists, 1820-1957"
"Nottinghamshire, England, Extracted Parish Records"
"Pallot's Marriage Index for England: 1780 - 1837"
"Reports of Deaths of American Citizens Abroad, 1835-1974"
"Selected U.S. Naturalization Records - Original Documents, 1790-1974"
"Shropshire, England, Extracted Parish Records"
"Surrey, England, Baptisms, Marriages and Burials, 1538-1812"
"Surrey, England, Marriages, 1754-1937"
"Sweden, Indexed Birth Records, 1860-1941"
"Texas Death Index, 1903-2000"
"Texas, Death Certificates, 1903–1982"
"Texas, Select County Marriage Index, 1837-1977"
"The Houston Jewish Herald-Voice Index to Vitals and Family Events, 1908-2007"
"U.S. and International Marriage Records, 1560-1900"
"U.S. City Directories, 1821-1989"
"U.S. Naturalization Record Indexes, 1791-1992 (Indexed in World Archives Project)"
"U.S. Public Records Index, Volume 1"
"U.S., Find A Grave Index, 1600s-Current"
"U.S., Headstone Applications for Military Veterans, 1925-1963"
"U.S., Social Security Death Index, 1935-2014"
"U.S., World War I Draft Registration Cards, 1917-1918"
"U.S., World War II Draft Registration Cards, 1942"
"UK, Incoming Passenger Lists, 1878-1960"
"UK, Outward Passenger Lists, 1890-1960"
"Web: International, Find A Grave Index"
"Web: Louisiana, Find A Grave Index, 1700-2012"
"Web: Lucas County, Ohio, Blade Obituary Index, 1970-2010"
"Web: Obituary Daily Times Index, 1995-Current"
"Web: Texas, Find A Grave Index, 1761-2012"

[tatewise]

From your other descriptions it sounds like FTM uses RESIdence attributes instead of CENSus.

The potential data loss can probably be avoided by converting FH CENSus events to/from custom Census events, and perhaps FH RESIdence attributes to/from custom Residence events, or something along those lines.

If that list is synonymous with Sources then they all use Method 2, because they are all composite lists of many source documents.