* Ancestry, FTM, FH and workflow

[Nick-V]

Rather than clutter the KB, here is brief feedback on "Fact tags" other than DATE, PLAC, Value and NOTE like email, web and phone. Given the plugin's default FTM options we get the following. Much as the options would suggest:

Code: Select all

0 @I4548@ INDI
1 NAME 0 Fake person for source citations /?/
1 EVEN Contact Value
2 TYPE Contact
2 DATE 1 JAN 1900
2 PLAC Contact Place
3 SOUR @S844@
2 PHON Phone Number, Web Site
2 EMAIL Email address
2 NOTE Contact Note
3 CONT Attribute Value:	Contact Value
3 CONT Place Details:	Contact Place
3 CONT Address Data:	Contact Address

We've not tested Fact tags before but FTM rejects EMAIL on import and PHON (with web site appended) gets moved to the note in an untidy manner (not formatted like yours). In the KB I raised the matter of Age and of course there are a handful of other such Fact tags. It seems to me the plugin doesn't need options for Fact Email and Fact Website (for FTM purposes)...what it needs to do is move ALL Fact tags that other than those with fields (DATE, PLAC and ADDR/Value and NOTE) to the Fact Note in a formatted manner.

EDIT: A similar thing is true for Media tags...separate options are not necessary, we have to move all tags to the Note unless they have a specific FTM field. They don't! The Caption works but as leant previously, the Date, Categories and Description fields in FTM appear to be unusable.

[tatewise]

Nick, when programming a Plugin there needs to be a precise definition of what is required.
Saying, "What it needs to do is move ALL Fact tags other than those with fields ... to the Fact Note" isn't helpful.

First of all, almost every Fact tag does have a field, so that means almost none will move to the Fact Note.
Please be more specific about what tags to export and what to move to Fact local Note.

My promise on Fri Mar 13 was:

I will happily update the Export Gedcom File plugin to export FTM compatible Gedcom, plus BOTH full & part images as discussed, and any other simple adjustments that may be required, but stop short at this stage of taking the round-trip recycling of data any further.

I am getting less impressed with FTM Gedcom capabilities by the day.
The Plugin was designed simply to convert FH custom tags and perhaps a few others.
Wholesale translation of large numbers of tags is getting tricky and starting to impact run-time performance even for non-FTM modes.

Perhaps it is time to rethink the round-trip concept or put it in abeyance until we understand more.

We now know that every INDI, FAM, NOTE, SOUR & OBJE Record type can have a Record NOTE.
(REPO could use ADDR, and SUBM & SUBN need investigation)
So perhaps a better approach would be to only export essential data and put the Record Id in the Record NOTE, etc.
Then on return trip, a plugin could restore the original Record Id and merge with the original FH Project.

[Nick-V]

Mike, I am following your lead in suggesting use of the note for all data that FTM has not got a specific field for and agree this is the simplest way. Remember my main input is researching FTM and trying to tell you what it has and has not got in the way of fields. As we know, FH's other data has to go in a note assuming we want to preserve it. Sure, "the rest" needs defining by someone probably by using the approach you suggested previously, expanding and right clicking each record type. So my statement is correct and it's not yet explicit. Lets be fair, you definitely know FH's fields far better than I do! We don't have to include them all....my prototype approach was to develop as the demand for certain data arises. We've gone further...

To clarify, the Fact fields in FTM appear to be few: Date, Place, Description and Note. Yes objects and citations also get attached. What I mean is that all other FH Fact (event/attrib) fields like Age have to go in FTM's Fact Note as most of them already are. These can be listed...but lets first clarify that we are saying the same thing conceptually. Perhaps we are getting hung up on the word field?

I'm not going to defend FTM. It is more basic than FH, does not stick to the spec etc etc...but despite any frustrations you have nearly completed a routine that, as far as I can see, puts all key FH data into FTM in a manner that can return to FH. I really do understand it is harder to take an existing program (the plugin) and add new features that bend it's processing than to create one from scratch pre-armed with the facts. I don't foresee any further unusual processing do you?

And lets remember...the benefits of using Ancestry web hints and updates on FH tree data cannot be understated.

The idea of storing a unique key for merging later is something you mentioned before. It is an alternative to total replacement. The jury is still out. What I have failed to do is resist the temptation to drill into the export detail and stick to my original plan of proving the concept at a higher level. No matter, although not proven I think we are both confident of the round trip. And although we've gone for detail on the export...we've nearly done it all haven't we?

Please remember I have a very short experience of GEDCOMs, FH, FTM and this whole topic. I never set out to provide a detailed spec for programming. But I have documented my findings as they occur, proposed and taken advice on solutions and now we've got something good done...it's not perfect but its all new, there was no published spec for FTM, no data map between the two products...there was a blank page and its now 90% full! I think we've done well together and I'm sorry for the frustrations...

[tatewise]

Having had a closer look, it appears the Plugin has rules for many of the other Fact tags already, and just needed enabling for FTM. The few remaining were easy to add.
Thus _EMAIL, _WEB, PHONe, TYPE Descriptor, AGNC Resp Agency, _SENTence Template, AGE, HUSBand AGE & WIFE AGE all become a labelled Fact Note.
The one remaining I am not sure about is CAUSe?
You don't mention it as an FTM field, but you have mentioned it in connection with DEATh and DIVorce.
What is the required export translation?
a) Always keep the CAUS tag line?
b) Always move the CAUS value to a labelled Note?
c) Do a) for DEAT & DIV and do b) for the others?

[Nick-V]

Oh that's good news then..a couple are new to me.

As a start, I would always put CAUS in the note. (B)

For DEAT we should keep the tag line (so A and B). When the tag exists a CAUS event is created by FTM but on export I believe FTM puts it back as a CAUS child of DEAT. The tag line is OK and I presume FTM takes this exception action for hint reasons.

For DIV and all other facts we should delete the tag line (so B only). When a tag exists, CAUS is appended to the Description ("div address; div cause"). The tag line is not OK.

Although I don't know all of FTM's quirks and there is no easy way to find them, I am "trying" to avoid FTM moving data around (using a note, appending to a field, etc.). I am trying to ensure data goes only where we put it and use of a note is always subject to our formatting standard.

[tatewise]

I have updated the Plugin V1.7.8 date 04 Apr 2015 in the KB Step 2.

This has the revised method of copying Source Media to Citation Media that is not only much simpler to implement, but also retains any subsidiary details, and supports all the Multimedia conversions of the Plugin.

It has all the Fact tag changes recently discussed.

I should mention that this recent development of the Plugin has involved more intensive testing than before that has revealed minor bugs in various export options not just for FTM but other target programs too. So thank you for that motivation which should benefit others in the long term because the Plugin is more robust.

[Nick-V]

Glad to hear that being a pain is also motivating

I'll get the new version and work thru. I'll try to check which fact tags now appear using recent posts and KB step 2 as a guide and for feedback.

I'm quite certain this plugin and the others are much appreciated even if people don't understand fully the effort you have put in. This long thread provides an insight !

[tatewise]

Some time ago it was mentioned that FTM only supports a single folder for Media files, and structure sub-folders were not possible.

The posting Importing FTM14 - lost media links (12488) suggests that sub-folders are feasible.

[Nick-V]

I can't seem to edit KB Step 2, locked by tatewise but the 15 minute limit never ends....?

[Nick-V]

Interesting, either I am wrong or this is the cause of his problem...he should open the GEDCOM and check paths to be sure...

WRONG: So far, all I can see is one can "link to existing media" or "add new media". The former is like FH. The latter (also similar to FH) lets you select a file from any folder but must copy that file to the "\{treename} Media" folder because that is what the link displays and GEDCOM contains. Or am I going mad?

EDIT: Yes you can have different folders. Another hidden gem tho. Still investigating but when you add media it is as above. However if you want to find missing media the correction facility allows you to COPY or LINK. Clearly linking only when you have a problem is strange so I'm now looking for how people link when they first attach media...

[tatewise]

Sorry, I've been updating KB Step 2. If keep refreshing and editing the time-out gets extended.

The options are side by side in pairs, because FH V5 has no Witness Role (2 _SHA%u) and Place Record (0 @P%d+@ PLAC) options, which if not side by side would disrupt the layout and make the online Help & Advice screenshot very different for V5 and V6.

Next version of Plugin will move the NICK & _USED options up to 2nd row below _LIST and _ROOT options.

The _FLGS custom events will use 1 EVEN Y which is better Standard Gedcom.

Regarding Media folders:
FH allows any Media file to be copied into any Project Media sub-folder when creating a new Media Record, thus maintaining a folder structure. We spoke about this earlier. FH even allows an external Media structure & Gedcom to be imported into a new Project and maintain the Media structure.

[Nick-V]

OK thought there was a KB lock problem...will continue with this media link investigation, then complete the testing and editing with KB then the new posting.

[tatewise]

In that other posting the 'new' user says all the Media files are listed OK in Tools > External File Links without any X errors, so they must all have imported correct paths via the Gedcom.

It is simply that he has not found where they are attached to Records, possibly because they are attached directly to Facts and Citations rather than Media tabs.

[Nick-V]

OK understood and very possibly true...

...I've noted at the base of KB Step 2 what we currently know and what we need to investigate before solving. I'm going to tie up loose ends first.

[Nick-V]

The presentation order and ideas for merging certain options are certainly tidier as far as an FTM export is concerned but this is a general export and I understand the compromises. Actually, we are very nearly telling users to take the FTM 2014 defaults on Tab 2 (PLAC records being the exception for now), you might even disable the Tab for FTM.

Yes the _FLGS Y action is good.

[tatewise]

I don't like the idea of disabling the Extra Options tab.
Remember this is a general purpose Plugin, and users may want to export to FTM, but if not interested in the round-trip may wish to remove some tags entirely.
Also didn't you want the option for NICK/_USED to use "" or ALIA?

In next version Record Website (1 _WEB) will Append to E-mail.
Also Media Format (1 FORM) will Move to Record Note.
(I presume FTM Media Record import rejects the FORMat tag?)

You seem to have forgotten our recent detailed discussions about the logic of FTM Fact Description rules. See this thread's postings dated Tue Mar 31, 2015 after Noon.
Summary:

• Standard Attribute Facts have Description = Attribute value but not in Note to avoid duplication,
  and Address Data: in a labelled local Note.
• Standard Event Facts have Description = Address 1st line and Address Data: in a labelled local Note.
• Custom EVENt Facts have Description = Attribute value for FH custom _ATTRibs.
  Custom EVENt Facts have Description = Address 1st line for FH custom EVENts.
  Both have Attribute Value: and Address Data: in a labelled local Note.
• Standard Attribute RESIdence has Description = its own style of data and Address Data: in a labelled local Note.
  RESI is also an oddity in Gedcom because it is an Attribute with no value allowed.

In summary Address Data: is always in a labelled local Note.
Events always have Description = Address 1st line.
Attributes always have Description = Attribute value.
Custom EVEN/_ATTR have Attribute Value: in a labelled local Note only so FH can reconstruct on return.

[Nick-V]

Yes I suppose disabling may be too radical. Yes it's true I suggested the "" vs ALIA option. I can never decide to be hard nosed or easy-going ! The option is a good idea because the name may get too long.

The two additions sound good...without checking let's assumes FTM rejects everything it has no fields for.

I had to read it several times but can see what you mean. I agree what you have done works and fits those words. However, I think the standard attribute Value needs to go in the note just like a custom attribute Value. We've both attempted to document this with some confusion - for my part I am simply tired and working at speed.

Duplication is a concern whether it be a custom or standard attribute, or a custom or standard address. I consider we should put all data in a note for consistency, no doubts. I now remember we will deal with duplication inconsistencies in the Import.

[Nick-V]

I now understand how you wish to preserve list data. Create a note containing references to list members and attach each member to this note. A couple of questions and feedback:

1. There is only one 0 Note display field in FTM so all 0 NOTES are displayed together which is untidy (see below).
2. One big note record is exported! However, data is preserved.
3. What if renumbering occurs - I can't remember what we learnt.
4. Is there a way to not have to attach the list note to every list member?
5. I wonder why the "@N22@" gets displayed
   ...thinking tomorrow...

I've created a little test data. Here is an excerpt from the plugin export:

Code: Select all

0 @I3525@ INDI
1 NOTE @N22@
1 NOTE @N23@
1 NAME Ian Gerald /Smith/
1 SEX M
1 BIRT
2 DATE 3 DEC 2010
2 PLAC Zdroje, Szczecin, Poland
2 NOTE Place Details:	Zdroje, Szczecin, Poland
1 FAMC @F31@
1 NOTE Local note 1
1 NOTE Local note 2
1 NOTE @N13@
0 @N22@ NOTE Named List:	Bookmarks
1 CONT INDI:	[1]	Gerald SMITH
1 CONT INDI:	[3525]	Ian Gerald SMITH
0 @N23@ NOTE File Root:	[3525]	Ian Gerald SMITH

Here is what the Note field displays - just one big note

Code: Select all

Local note 1

Local note 2

Named List:	Bookmarks
INDI:	[1]	Gerald SMITH
INDI:	[3525]	Ian Gerald SMITH

File Root:	[3525]	Ian Gerald SMITH

@N22@

THIS IS AN FTM NOTE RECORD N13

Here is what is exported - one big note:

Code: Select all

0 @I1@ INDI
1 NAME Ian Gerald /SMITH/
1 SEX M
1 BIRT
2 DATE 03 DEC 2010
2 PLAC Zdroje, Szczecin, Poland
2 NOTE @N1890@
1 NOTE @N1889@
0 @N1889@ NOTE
1 CONC Local note 1
1 CONT
1 CONT Local note 2
1 CONT
1 CONT Named List:	Bookmarks
1 CONT INDI:	[1]	Gerald SMITH
1 CONT INDI:	[3525]	Ian Gerald SMITH
1 CONT
1 CONT File Root:	[3525]	Ian Gerald SMITH
1 CONT
1 CONT @N22@
1 CONT
1 CONT THIS IS AN FTM NOTE RECORD N13

[Nick-V]

Mike, I see you've taken time to respond carefully to several areas in the KB Step 2. Your explanations and some sleep make it easier to focus so I've edited the KB a little where its easy, and where more words are required I've replied below to seek your thoughts.

Use of Description field for Address and Value etc.:

1. On RESI, I understand we can let Ancestry add its Description and consider what to do later in the Import so no action.
2. Thanks for reminding me we need to differentiate between events and attributes in the Import which is why an exception was made for standard events. It would be good if it were possible to address this another way and retain consistency. Could something be added to the Note to differentiate?
3. Avoiding duplication (address or value in both fields and note), I now remember we decided to allow duplication (rather than any export options) and resolve inconsistencies in the Import so no action.

Lists and Preserving PLACe record data:

1. There are instances where certain is more difficult to preserve in FTM and an elegant solution requires consideration. Creating a Note Record for Lists then connecting it to each list member causes the single record note for that person to be severely cluttered. Indeed, FTM then merges any incoming 0 NOTEs and exports as one. I wonder about the feasibility of connecting the 0 NOTE just once to a fake Source or some other entity to get it our of sight (also I can't remember if renumbering of INDIs compromised this).
2. Place records are output as Sources to preserve Place data but they intermix with genuine sources confusingly. If this turns out to be the only solution (not NOTEs or otherwise) we might at least prefix them so they sort to the end.
3. This type of principal might also be adopted for other data (REPO fields?) if the need arises.

To/From place

1. Are you able to confirm if IMMI and EMIG the only facts (events or attributes) that can have a second place?

Family Census events

1. I confirm previous testing that these are ignored on import whereas INDI, CENS are accepted. A custom event appears necessary to preserve FAM, CENS. I presume FTM's design logic is that Facts are either for an Individual or Shared but cannot be both. This controls where they are presented on screen.

[tatewise]

Use of Description field for Address and Value etc.:

1. RESIdence Attribute is a special case in Gedcom, FH & FTM/Ancestry so agree no action on export.
2. Custom _ATTRibute is a special case. For consistency with Standard Attributes agree no labelled Note for its value. To differentiate from custom EVENts that may have ADDRess in the Description field, I propose an unusual prefix such as _ATTR: on its value in the Description field, i.e. 1 EVEN _ATTR: Value...
3. All Events have ADDRess 1st line in the Description field and full Address Data in local Note so agree no action.

Lists and Preserving PLACe record data:

1. Named Lists
   May I suggest that before proposing a solution you investigate its feasibility in FTM. What happens to single or multiple Note Records linked to a fake Source (or other Record type)? Does such a fake Source & Notes survive round trip (I thought a Source had to be linked to something)? Renumbering of all Record types (INDI, FAM, NOTE, SOUR, etc...) is a problem, because every Record type can appear in a Named List.
   My first reaction is, since you don't use Named Lists, leave alone for now, but document for later. FTM is *** awkward!
   Afterthought: Maybe use a Named List SOURce Record instead of Note Record, and link that to every Record in list &/or to a fake Record that survives round-trip? What are the Record types that survive without being linked to anything?
2. Place Records
   Not keen on Note Records linked to Place tags because FTM converts Place local Notes to Note Records so if there is also a linked Note Record derived from Place Record they may get horribly merged?
   Prefix idea is a good one. What leading character do you suggest that will sort to the end of FTM Sources?
3. Repository Records, etc
   Yes, a Source record related to Repository, etc, is a good idea, if we can resolve the renumbering of Records.
4. Record Renumbering
   I suggest the answer is to put the FH Record Id somewhere in every Record. Perhaps use a labelled local Note where possible, otherwise (for NOTE, REPO, SUBM, SUBN, and some SOUR) prefix every Name/Title with Id in say square brackets [S1234] in the style of FH? This could also act as the Prefix you suggested above? All synthesised Sources will have higher Id numbers so will automatically sort to the end? Those records would only sort in Rec Id order and not by Name/Title ~ is that a problem?

To/From place

1. Confirmed that _PLAC is only in IMMI and EMIG.

Family Census events

1. OK, no problem, custom Event for FAM CENS Event.

[Nick-V]

Use of Description field for Address and Value etc.:
2) What I am proposing is that the note always contains the address and value for consistency. We regard the note as master data not the Description. To differentiate, we put an indicator (such as _ATTR) in the note not in the Description (where it would appear unpleasant). I think this functions fully and presents well?

Lists and Preserving PLACe record data:
1) Named Lists
Yes further checking of note handling is necessary if the principal is an acceptable alternative to use of Source records. Both alternatives (forget it or use a linked Source) are good in principal and the latter can be investigated before we decide together with establishing which record types survive. I'll put some hours aside presently.
2) Place Records
Of course the user can still opt to not export these. But if they do export the prefix might be "{I'll do some thinking and testing including [P124] and edit this and the KB.}".
4) Record renumbering
I need to refresh myself on exactly what this does and does not affect before resorting to storing the numbers anywhere. For discussion purposes, if we exclude Lists and we exclude using a number during an import merge can you help me with what else we need the number for?

All the rest good, many thanks. I'm now looking at media folders for a while...

[tatewise]

Use of Description field for Address and Value etc.:
2) OK, labelled Note for Value in every Attribute, including if the value is blank for custom _ATTR to act as a differentiator.

Lists and Preserving PLACe record data:
1) & 2) Await your investigations.
3) I cannot exclude Named Lists from Plugin as other users may want them whatever you prefer.
Record numbers are also needed to reconstruct Witnesses from Notes, and ideally Place Records from Sources/Notes, and relate Repository Records to their associated Source/Note for tags without FTM fields, et al...

[Nick-V]

PLACe records...thinking aloud.

1. The objective is purely to preserve FH data because FTM has no Place record. The FH data to preserve is Place name (which matches with the place tag in a Fact Note for example), lat/long, STAN and links/tags for media and notes.
2. While FH has place records, I understand it has no links to these. The name is simply the basis of a "database" used for normalisation. If true, there is no need to create links in FTM either, we simply want to store and re-export some place data together with its links/tags for media and notes.
   
   Just before we continue ahead with Source Records, can we think of any other Record or tag (e.g. 0 NOTEs or any other tags that allow child notes and media) that we might consider if links are not required?
   
   Prefixing with [P1234] would sort places together in the list of Sources but they would be at the top and not alphabetised. Use of { | } or ~ do not sort properly either. I think the answer is "ΩPlace: ,,London, England" (note the preserved leading commas).

[Nick-V]

Use of Description field for Address and Value etc.:
2) I forgot valueless attributes also needs a line in the note! Perhaps the Attrib/Event indicator in the note can be subtle. I don't know...maybe a 1 char prefix like "~" or wrap the heading in {}...just a minor preference over putting ATTR or whatever...whatever you decide, go ahead...

Lists and Preserving PLACe record data:
3) I accept we need a solution to handle Lists and this and indeed all of our recent work on other fields etc. is for others - my own data has been in Ancestry for some time and hint matching is really good! I must admit I really need to investigate and understand Witnesses and indeed Subscribers etc. soon! However, I remain uncertain about the need for links to PLACe records, and perhaps SOURce links to their REPO is not an issue (mentioned in the post above). Let's rest this just for now...

[tatewise]

Sometimes I feel I am going round in circles with these discussions, Nick.
Perhaps I don't explain myself clearly enough, so will spell it out in more detail with examples.

Use of Description field for Address and Value etc.:
2) We have agreed a labelled Note for the Value in every Attribute, YES?
In a Standard Attribute with no Value there will be no labelled Note, because we know which Facts are Events and which are Attributes from the Gedcom specification.
In a custom EVENt derived from a custom _ATTRibute with no value, there will be a labelled Note with no value, i.e.
Attribute Value:
The label itself will act as a differentiator between custom EVENts from custom EVENts and those from custom _ATTRibutes.

Lists and Preserving PLACe record data:
3) If we don't preserve Named Lists, Place Records, etc, in SOURce Records in FTM, what are the options in FTM?
That is why I asked "What are the Record types that survive without being linked to anything?"
There can't be many choices.
Presumably we would not want to use INDI or FAM records.
HEAD, NOTE, REPO, SUBN, OBJE do not support linked OBJE media, so unsuitable for Place Record details.
That leaves SOUR and SUBM. Does FTM support SUBMitter records?

If we have to use SOURce Records, then I am sure you said they must be linked to something to survive round trip.
Then why not link SOURce Records holding Place Records to their Place tag?
Otherwise we are into creating fake records just to link SOURce records to.

I forgot to say the leading character chosen to sort records must be an ANSI character that is compatible with FH V5.
Therefore Ω is unsuitable, because the Plugin cannot use it, since the Plugin runs in ANSI mode.
So the choice of characters is limited to those on your keyboard plus a handful of other symbols & accented characters:
!"#$%'()*+,-./:;<=>?@[\]^_`{|}~€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—™š›œžŸ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿

Don't get hung up about Witnesses.
They are just like Father, Mother, Child links to other Individual Records.
But they also have associated Role, Note, Source & Sentence tags.

Another feature that also needs preserved INDI Record Id is the ASSOciated Person link.
They may also be needed for the HEADer and SUBmissioN records.
So I think there is no escape!