* Ancestry, FTM, FH and workflow

[Nick-V]

Sadly the answer is as described best by the data below. In summary:

Citation text is neither recorded nor output although such text can be entered manually. Each appropriate event for an individual has links to both a) the one 1911 Census SOUR source record for all families in England that year and b) the one OBJE image of the particular census for that family. Ideally any citation text would therefore be stored against the OBJE but its not (as should the PAGE data).

As with the Ancestral Data plugin a source for each family would have been better. I suppose we could create that and then re-merge on output.

One could live without citation text considering the linked data available and the image. However, I wonder 1) can this data be stored and re-output from FH and 2) how might one view, print and web an image of the census?

0 @I19@ INDI
1 NAME Matilda /Ridgway/
2 SOUR @S10@
3 PAGE Class: RG14; Piece: 10332; Schedule Number: 36
3 OBJE @M2547@
1 SEX F
1 BIRT
2 DATE 27 SEP 1880
2 PLAC Marylebone, London, England
2 SOUR @S10@
3 PAGE Class: RG14; Piece: 10332; Schedule Number: 36
3 OBJE @M2547@
1 RESI Marital Status: Married; Relation to Head of House: Wife
2 DATE 02 APR 1911
2 PLAC West Mersea, Essex, England
2 SOUR @S10@
3 PAGE Class: RG14; Piece: 10332; Schedule Number: 36
3 OBJE @M2547@

0 @S10@ SOUR
1 AUTH Ancestry.com
1 TITL 1911 England Census
1 PUBL Name: Ancestry.com Operations, Inc.; Location: Provo, UT, USA; Date:
2 CONC 2011;
1 REPO @R25@

0 @M2547@ OBJE
1 FILE C:\Users\john.smith\Documents\Family Tree Maker\Test2 Media\1911 England Census.jpg
2 TITL 1911 England Census

[NickWalker]

Hi Nick

As with the Ancestral Data plugin a source for each family would have been better. I suppose we could create that and then re-merge on output.

Consequently, perhaps all census sources and facts and images (painstakingly put in using the Ancestral Facts plugin) should be deleted before uploading to FTM

Firstly I think you are referring to my Ancestral Sources software and it isn't a plugin, it is a separate program to Family Historian although it interacts with FH quite well and there is a plugin available which launches it.

Secondly, Ancestral Sources does have various options to change the way you record your data. So, for example, the 'recording method 2' allows a single source for census year with details to be recorded in the citation. It is also possible to have the transcriptions held in a note record linked to the citations. Although these are not recommended for FH users, they may be more compatible with what you're trying to do with FTM (but I've not really read your posts in detail as this isn't an area of interest for me).

Cheers

Nick

[Nick-V]

Thanks Nick

Yes, "Ancestral Sources", sorry, it does launch like a plugin...good program!

Thanks for confirming AS can optionally treat sources the same way as Ancestry (one per year instead of one per family). You have identified a potential solution, I wonder if we could create and use a note records for each family census somehow (reverting to the original structure on output).

Given the view and web output requirement I still suspect we still need to duplicate source records (as AS option 1) a note record might help in reconstructing.

[Nick-V]

To provide easy access to the latest thinking, the key information from this thread has been compiled into a single Knowledge Base article at how_to:exchanging_data_with_ancestry_ftm|> Exchanging Data with Ancestry & FTM.

This thread remains the place to discuss this subject. The KB article will be updated as new ideas arise.

[tatewise]

Comments initially on how_to:exchange_with_ftm:step_1_prepare_your_fh_data_for_exchange_with_ftm|> Step 1 Prepare your FH data for exchange with FTM especially understanding what does and does not import into FTM. The way the samples below are generated is by open the All tab of Property Box and using right-click on first the title and then each subsidiary field to create as many different items as reasonably feasible.

Custom Sentences ~ Not sure why mentiond as Export Gedcom File Plugin deletes them.

Media Records have the following standard format, can you confirm what does and does not import into FTM, especially the Record Id O123 or does it become M123:

Code: Select all

0 @O123@ OBJE
1 FORM jpg
1 FILE C:\Users\1911 England Census.jpg
1 TITL 1911 England Census
1 NOTE @N1@
2 SOUR @S36@
1 NOTE Any note text.
2 CONT Can be on more than one line.
2 SOUR @S36@
1 REFN Custom id 123
2 TYPE Type of id

Repository Records have the following standard format, can you confirm what does and does not import into FTM, especially the Record Id R123:

Code: Select all

0 @R123@ REPO
1 NAME Repository name
1 ADDR Address in full
2 ADR1 Address line 1
2 ADR2 Address line 2
2 CITY City
2 STAE State
2 POST Postcode
2 CTRY Country
1 PHON Phone number 1
1 PHON Phone number 2
1 EMAIL Email address 1
1 EMAIL Email address 2
1 NOTE @N1@
1 NOTE Local note.
2 CONT Can be on multiple lines...
1 REFN Custom Id 123
2 TYPE Type

Family Records ~ Similar to above including more Events & Attributes.

Source Records ~ Similar to above.
It might help to separate the Census Source scenario from all other Event & Attribute Sources.
I do not understand your comment "(clutter which is not an event anyway)".

Note Records ~ Similar to above.

Individual Records ~ Similar to above... I know you have checked many of these tags, so they don't need repeating, but there are many other tags.

This may be revisited when I examine Step 2 in more detail.

[PeterR]

Perhaps I'm missing something, but I don't understand why Step 1 is described as "one-off". Surely, if new data are subsequently added to FH, it will be necessary to repeat Step 1 in full, unless somehow the user can guarantee that all new data are restricted to the FH/GEDCOM subset that is fully compatible with FTM.

[Nick-V]

Peter, if new data is added incorrectly after tidying up yes it'll have to be done again. If proceeding manually don't enter stuff you know to be an issue. If automating we need to know what to avoid.

The solution today is to make your data as good as possible and use existing tools. The future solution may be automated and control such issues.

[Nick-V]

Mike, thanks for taking time to study my evolving waffle...

I'll get to these matters shortly. My current focus is proving the spec with a VBA prototype therefore the spec is being kept up to date. You'll note the T marker as each requirement is proven or elaborated on. It's simply too complex to spec in one large step...this step by step approach of actually doing it is progressing well.

I'm continue to gradually becoming more confident that an FH export, merge into FTM, update in Ancestry, export from FTM and replacement of FH data can be made to work. The provisos include the object records prefixed on export (re-prefixing has to be avoided somehow) and how complex ones use of FH currently is (no automation will do everything). A round trip is very likely to work but there will likely be reasonable compromises. For me, Ancestry's large volumes of hints and updates (data and images) is the trump card.

In the meantime, we know we prefer (not need) all (part and full) images output as object records with actual paths, and full images need links to all INDIs from which they arose. This will not change.

[tatewise]

That last point is one I meant to tackle earlier.
If we are only interested in collecting the result of hints from Ancestry/FTM, why do Media images need to be exported to FTM at all?
I do not understand the benefit of both part & full versions, unless they in some way help with hints.

By comparison, the MyHeritage interaction in FH does not rely on images at all to gather hints.

[Nick-V]

Yep FH images exported to FTM are not required purely for Ancestry hints. However the data updated in Ancestry has to get back to FH again and this includes OBJE links sent to Ancestry. Further, putting images in Ancestry (the least of our issues frankly) provides Ancestry users with the fullest access to tree data...and images are important. Finally, Ancestry gives us images of vital records and these belong back in FH. Wouldn't it be great to say FH can go boldly where no "person" has been before !

Why part and full images? When you create an FH web site you can click on the portrait of an individual (part) and up pops a page with the whole family shot (full) that it was clipped from. I suspect the closest we can achieve is linking both the part and full images to the indi (or indeed family). Of course, users may not want full images (option) because they are nasty before being cropped or contain sensitive backgrounds like grandma with her pants down.

The _PHOTO tag works a dream BTW.

[tatewise]

But if we don't send Media to Ancestry/FTM, then the OBJE links don't do the round trip, and all the problems with prefixes, absolute/relative links, etc, melt away. Alternatively, the returned Gedcom can have the FH Media stripped out, especially easy if FH never uses Citation Media, and Ancestry/FTM only uses Citation Media. When the returned Gedcom is merged back into FH the orginal FH Media that remained in FH will be retained.

I suspect it is unlikely that all the Project/Gedcom data can remain intact throughout repeated round trips.

[Nick-V]

Yes much of the GEDCOM data will do the round trip although there will be issues like info now in notes fields and I can't talk for those who use FH's data in a more complex manner.

My current thinking gets rid of the need to merge back into FH. Merges are hard work and data will get out of sync due to the need for manual intervention. We already have 2 - FH to FTM (which looks OK so far) and Ancestry to FTM.

Getting all the good stuff back to FH from Ancestry including citations and images is important. Getting FH images and links into Ancestry is a benefit and works already as does getting images out out.

Yes, the re-exporting from FH of object with prefixes needs thinking about - for now my head is elsewhere and I hazard a guess there is a solution. I don't see any others issues with objects and their links at all. Given what the BOTH option already does all that is needed is the links which could even be added in by a small routine post export plugin.

I have done round trips with full replacement of FH data...it works but there are some matters to automate just like the export.

I'm playing with the data, identifying differences and on some detailed steps to resolve them. Is it relatively straight forward for the BOTH option to create the multiple links to the full images?

[tatewise]

As you know, the BOTH option has multiple links to full images for local Media Objects already. There is no problem with BOTH option multiple links to Media Records linked to full images, because they already exist. The more complex job is linking to newly created Media Records linked to the new part images, but I think it is feasible.

I don't doubt that round trips are possible with certain subsets of the data, but we need to thoroughly understand exactly what those subsets are by detailed investigation, otherwise other users who rely on the automation tools will too easily violate the rules if the tools are not thorough.

If the rules are well defined, and the Record Id remain consistent over the round trip then the merge may well be automated by Plugin. That is why I am interested in how Record Id are handled by the FTM import process.

[Nick-V]

Mike, my head is spinning (tired)! I don't understand "multiple links to full images for local Media Objects". Don't worry, I'll create a BOTH file and look again at what's there. When I last looked I though the part images are attached to the INDI (LMOs?) and the full image are OBJE record with no links to anything yet (which was a bit odd!).

Yes I understand creating the new links to the full image (the 0 prefix record) might be "interesting"...I'm not quite there yet!

Yes I am trying hard to be very explicit about what data is working. When (hopefully) my simple data works we can move forward to other data. I suspect the tools will take a while to be "thorough". I suspect there are issues in many established systems. I'd hate to allow an aim for perfection stop me at least getting started!

Record IDs change. I have gaps in my FH numbering probably from deletions. It appears FTH resequences. The merge into FTM is not reliant on such numbers, which after all are just for the computer to link things. The FTM merge matches on name, dates and other factors, you set a score and must connect manually any falling below that threshold. It works fine.

Surely, FH's standard merge is also intended to import data from different software and not rely on numbers. A nightmare to have to check it. I understand you are considering a custom merge based on the @numbers@. Not going to happen (unless a number translation with external table was also done). Still prefer to overwrite in the first instance. When we get a flow working for the average user improvements could be made...we need to understand the FTM and Ancestry basics yet !

If you want a copy of my data in both FH and FTM format (you specify exactly), I'm quite happy to email it for your use only.

[Nick-V]

Here is some BOTH data using my terminology (cross fingers):

' Below is the full image as a media record with a "0" prefix. There are no links to @O210@ at all so it can only be seen in a "media library", not against an individual or family.

0 @O210@ OBJE
1 FORM jpeg
1 TITL John Smith 1948
1 FILE C:\Users\...\Family Tree Maker\Test2 Media\0O210 0020.jpg
1 NOTE Keywords: picture keywords
2 CONT Media Date: 1948
2 CONT Picture Note: picture note
2 CONT

' Below is a part image taken from that full image but as an object attached to the individual - no links are required.
0 INDI
1 NAME bla bla
1 OBJE
2 FORM jpeg
2 TITL bla bla 1948
2 FILE C:\Users\...\Family Tree Maker\Test2 Media\1O210 0020.jpg
2 NOTE

So for clarity:
1) both objects import to FTM so no changes necessary however...
2) you mentioned all images might be media records in future (although it doesn't matter for this purpose)
3) INDI links "1 OBJE @O210@" need to be inserted. Several individuals might point to the same media record.
4) INDI tag "1 _PHOTO @O210@" would tell FTM the profile picture to use. Only to be created if it does not already exist.

[tatewise]

I think it is time to split this thread into different discussions focussed on specific aspects of the concept.

The Media discussion continues in Ancestry, FTM, FH and workflow ~ Media (12429).

The Sources and associated Ancestry/FTM hints issues are discussed in Ancestry, FTM, FH and workflow ~ Sources (12430).

The Data retention issues focussed on what other data will be lost or retained in the process are discussed in Ancestry, FTM, FH and workflow ~ Data retention (12431).

[Nick-V]

Good idea, understood. First post in thread updated to provide links.

[Nick-V]

Mike, I'm getting back to your posting above dated Thu Mar 12, 2015 4:13 pm. A quick reply because I need to shift from documentation to progress - please excuse tone and typos.

Thanks for letting me know a fast way of preparing test data for all FH standard fields. This will be particularly useful for anyone mapping all FH fields against FTM. My objective is a simple routine that can be added to incrementally - I currently recommend a strategy of getting something working over preparing a comprehensive specification of the entire problem with no assurance of any outcome. This project currently has just two resources. I acknowledge your interest in a complete solution...I just think we need steps.

For info, FTM can also export to FTM format files - I am selecting GEDCOM and UTF-8.

Custom Sentences: People might want to fix their accidental data errors in FH.

Media Records: The output from the export plugin creates @0???@ records and these import to FTM (few fields, not even date - investigate later). However, I didn't realise until you mention it but FTM exports @M???@ records and not @O@. I don't know the difference. I don't know what FH can import and suspect a renaming is straightforward.

0 @M3487@ OBJE
1 FILE C:\Users\......\Documents\Family Tree Maker\Test2 Media\0O458 GRG_0057.jpg
2 TITL Gerry & Gloria wedding 1959

REPO records in FTM have a Name, Address, Phone and EMail.

The story of others record types will be the same. I understand what you are seeking in the way of a spec and understand your have the return merge and complex FH users in mind. Today, there is no document that maps all FH fields to FTM - a massive task. There are far fewer fields in FTM so the reverse approach might be better!

Instead of working out what doesn't work I recommend we focus on what does work. In V1 people can constrain themselves or wait. V2 will add a few more priority fields etc...

If the objective is a fully comprehensive routine we need more resource. I recommend we take a more pragmatic approach and evolve a solution according to demand for it and demand for specific features. Lets face it many people are happy with FTM and don't need more.

[tatewise]

FYI: I changed the link to a date-time-stamp because the link landed in the wrong posting.

I don't think that KB page is the place to talk about user 'accidental data errors'. That is a completely different tome

On import FH appears to allocate new Record Id for @M???@ records in much the same way FTM does for @O???@ records. But what if we supply FH with @O???@ records and FTM with @M???@ records, by converting the letter en-route? Maybe then the Record Id number will be retained, and that might help synchronisation?

BTW: The level 2 TITL is invalid, and must be converted to 1 TITL on import to FH, so needs mentioning in KB.

I feel I need to understand what data will export to FTM, and what will not, otherwise users of my Export Gedcom plugin will blame me when it does not meet expectations, and some of their precious data gets lost in the roundtrip. The plugin needs to warn users, when they choose the special export mode for Ancestry/FTM roundtrip, what data will be lost, or even better refuse to complete the export if such data is encountered.

I appreciate you are on a steep learning curve, but if no other users become interested, you may also have to help with writing &/or testing Plugins too, especially as I do not subscribe to Ancestry nor wish to purchase FTM.

[Nick-V]

Thanks for the link change...the accidental data thing...its easy to make data inconsistent and to change things accidentally...one of the advantages over FTM is the various ways we can choose to employ in FH to run spot checks. SENT was not something I ever imagined having changed !

Still don't understand M from O but I get the idea and we can check. However, it does appear that FTM changes INDI and FAM numbers by the time they are exported. I suspect a "defrag" is done on import perhaps for efficient use of arrays.

TITL is used in three places in my data and there is already a note in the KB (import back to FH section) to investigate the exact situation. I've noted your point there.

Yes I support the need to understand. Let's not jeopardise the reputation of the existing export plugin and call this something else. Further, for now it is probably easier to say what it does do and explicitly say all other data will be lost. If we explain its version 1 aimed at FTM and basic FH users with provisos and for a specific benefit they can choose wisely. One day we can take away their need to think!

I'm flat out working on this at the moment and have to prioritise my efforts. I am documenting too much, testing and even having a go at a VBA prototype to prove things (despite total inexperience - yep steep curve). I am quite prepared that no users will become interested...this kind of supports the theory that simple is all that is needed. If users fail to engage now then a comprehensive solution is not likely to be of enough benefit to them. I didn't want to buy FTM either, but £20 was actually well spent to find a way to get my data into Ancestry better, and to provide an opportunity to learn and indeed deliver something helpful or even essential to my research.

I'll carry on noting my findings in the KB and try to get a routine doing the job in a rough manner. Once the concept is proven we can take it a step further.

[tatewise]

1) M from O
The Gedcom spec simply says every cross-ref @Id@ must be unique and every link to an @Id@ should have a matching record with that @Id@. The Id could simply be a unique number, but most software seems to make them unique by allocating an initial letter for the record type, followed by a serial number. In most cases the initial letter of the record type name determines that letter, I for Individual, F for Family, and so on are popular, but there are a few abnormalities.

Media uses the OBJEct tag, so should the initial letter be M or O? FTM chose M and FH chose O, and when they import their own lettered Id should keep the serial number. FH does, what about FTM?

Submitter SUBM records use U in FH.

Submission SUBN records use B in FH.

2) TITLe
The Gedcom spec sometimes uses the same tag with slightly different sematics.
1 TITL is used to define the Title of SOURce records and Media OBJEct records, but also the Individual TITLe attribute fact.

1 NAME is used to define the Name of REPOsitory records and SUBMitter records, but also the Individual NAME details.

[Nick-V]

Thanks for the additional info.

By now I can take fairly good guesses at the results FTM will provide. I have a little more understanding of their development motivations. Therefore, I guess they have decided to accept both M and O records as pictures (media or objects) but to use M as the primary. They did something similar with NPFX accepting both that and TITL with a preference for latter. And with NICK they went for ALIA on output (can't remember if they can import ALIA).

I also guess but will check that all imported data gets renumbered to remove gaps and all merged data gets matched on name, dates and other factors but not numbering. I will check by removing a record and exporting to check for renumbering.

Bottom line is we need to think simple. It is a basic product and that's what we are working with. One might admit a full replacement round trip is only feasible if you can constrain yourself to FTM's data. Good definition for stage 1.

Stage 2 therefore is if you wish to retain FH data that cannot be stored in FTM...the solution involving therefore a merge back to FH.

Working to the lowest common denominator (FTM) is a perfectly reasonable scope for a stage 1 project. All FTM users work with it.

[Nick-V]

Numbering in FTM appears much as suspected. When I delete a person in FTM the export leaves a gap where they were.

When I delete that person externally then import to FTM then export there is no gap...renumbering has occurred. BTW, I imported a file just exported from FTM after one simple edit.

Conclusion, FTM doesn't give a stuff about numbers. They are not a UID or even an IUD - they are relative links and it doesn't matter what they are. It renumbers on import probably for programming/memory reasons and does not renumber on output.

[tatewise]

I would happily go along with the concept of keep it simple, and focus on the sole objective of gathering hint related Sources from Ancestry, c.f. MyHeritage hints in FH.

So I still don't see where extravagant conversion of full & part frame images capable of being recycled through FTM and back into FH fits into that simple objective, especially as we are discovering that FTM is hindering that recycling of data.

Perhaps it is time to take a step back and review what we know. Can we discard the idea of round-trip recycling and just export from FH whatever FTM/Ancestry tolerates to allow hints to work. Then selectively export the hint related Individuals/Families and Sources from FTM for integration back into FH. If there is a way of somehow naming the Individuals uniquely (NAME, REFN, RIN, RFN, etc) then a plugin should be able to match them up and insert the new Facts and Sources.

[Nick-V]

Ancestry connects objects and updates data when you accept a hint. The objects must get back to FH. If full images are media records (as they are by accident) then adding the links makes it work. Yes text fields on images do not round trip...other than title I don't use them. Remind me where the hindering is - I'm juggling too many balls.

If I stop researching and documenting I am probably not far off having all of my data and images moving to FTM using your export and a VBA routine...

When the routine is done most of the conversion back should be manageable too.