* Plugin to sort problems with FTM Imports

[Barnowl]

FTM_Fixer was conceived as a One Stop Shop to deal with import problems arising on importing Family Tree Maker (FTM) data to Family Historian (FH). It is only expected to run once, on a tree imported into FH from FTM, preferably immediately after importing. This avoids any problem with the order of applying many ad hoc individual fixes where one may depend on another having been already run – or not run.

Its aim is to reduce the UDF Count to zero, which it now doing for my data

If an attempt is made to run it a second time it will not run.

You should use the option to include media, both when importing to FTM from Ancestry and when importing into Family Historian.

It does not attempt to fulfill all the requirements of a round-trip mechanism – at least not in this version - but will be of interest where a tree is being moved permanently, or where regular one way imports are performed in order to use FH reporting features.

Several of the problems it fixes seem to have been addressed in v6.1 which I have not yet looked at yet – I will make some adjustments to fit in with this in due course.

FTM_Fixer addresses the following problems with FTM data imported into FH
1. Truncation of longer records in Custom Events, OCCU, RELI, EDUC and PROP
2. Loss of Images from source citations
3. Loss of Military information
4. Loss of Employment records
5. URLs are lost. For now putting them into notes.
6. All Media described as Picture even if it is a Story
7. Default Picture information is lost
8. Address records stored inline in individuals is lost
9. Repository records lose Email Address. Address records are truncated
10. Media records are headed by their file name not their title
11. Many odd bits of HTML left in text – mainly &nsbp; & and ' – these are tidied up
12. Media references against an Individual NAME are removed
13. “Other Port” records (_ORIG and _DEST) showing the other end of a voyage can optionally be removed, converted or left alone.
14. “Bare Addresses” where Addresses records are stored directly under an individual are converted to RESIdence records.
15. Quarter Dates formatted as e.g. “Jan-Feb-Mar” or “Apr May Jun” are corrected.
16. Descriptions may optionally be trimmed to a user specified length to avoid over-filling small boxes.
17. Publication Info for Ancestry sources which is endlessly repeated can optionally be combined into a referenced note, simply discarded, or left as it is.
18. Source and Media records may optionally be given consistent parallel numbering. 
Source Images. The fact that source images are not displayed is due to FH’s inability to attach an image to a Citation, but only to a Source, meaning that to fix the problem you must embrace one of several Source Methodologies, and the data must be rebuilt accordingly.

1. Method 1 – Create a source for every citation – citation in “Where in Source” and “Text from Source” (Most sources – much duplication)
2. Method 2 – Combine sources as much as possible –citation in “TextFromSource” and “Note” within the source keeping it general to all instances. (Smaller increase in sources)
3. Method 1.5 - While keeping the more general sources of Method 2, put individual citations in the “Where in Source” and Text from Source easily visible at the front source form. (Same increase in sources as method 2)
4. Method 3. – Keep the extremely general sources of Ancestry / FTM. (A whole census is a single source). Do not attach media to the source at all, but store it directly in the individual. Store any information that would help future plugins to tie the two together. (Least sources – All media visible but links broken)

These methods are discussed in more detail on the import forum.

After a welcome screen (below right) FTM_Fixer offers the following options:-
1. Source Method
2. Record Numbering – Sequential, From Media Filename or None
Source and Media files can optionally be numbered, either sequentially or based on the Media filename. The source and media file are given the same name.
3. Convert long descriptions to notes
4. Number of words to keep in truncated descriptions
5. Number of words to keep from truncated line
6. Source publisher – Discard, In Note, or In Source
7. Other Port records – Discard, Restore, Leave

[img]FixerOptions.png[/img]

From that point processing is automatic. Progress through several processes is displayed on a double “Process Time / Overall Time” progress bar. At the end a report is displayed showing UDF count before and after processing and the number of errors of various types corrected.

[img]FixerResults.png[/img]

[tatewise]

That is certainly an impressive piece of work.

Have you looked at the recently updated how_to:import_from_family_tree_maker|> Import from Family Tree Maker (FTM) especially the General Advice and Quick Fix Plugins?

Also the updated how_to:handling_unrecognised_data_fields|> Handling Unrecognised Data Fields?

Could we discuss a co-ordinated solution involving:
(1) Dealing with UDF Fact tags.
(2) Using FH V6.1 new features.
(3) One Plugin to fix everything else.

[shoshk]

Hi Ian,

When I tried running your plugin, I received the message "Attempting to delete a null pointer OBJE[1]"

Any ideas?

Thanks,
Shosh

[Barnowl]

Hi Shosh
Thanks for trying my plugin.
This is a bit of a puzzle.
What is going on at this point is sorting the Media Title - promoting the TITLe from level 2 to level 1.
OBJE[1] means it is processing your first OBJEect
I believe you are familiar with plugins. You are looking at the function fix_media(). It has successfully written the new level 1 TITL and is trying to remove the incorrect Level 2 record.

Could you possibly post the GEDCOM records for the first Media Record - 0 @O1@ before and after running the plugin.

Are you running v6.1? If you are I am sorry but I have not yet modified the plugin to work with it. I think that may have already done this fix.
I have just tried changing TITL to level 1 in Notepad before running, and I get the same error as you.
While I could fix this I cannot guess what other problems there would be. Sorry if this is your problem.
I will of course be updating the plugin as soon as I am able - and also dealing with a wider range of UDF tags.

[shoshk]

Hi Ian,

Yes, I am running 6.1. So, if I understand you correctly, that's what's causing my problem. I'll wait for your fix.

Shosh

[JohnCulley]

Hi Ian

I tried your plugin and had a similar problem - "Attempting to delete a null pointer OBJE[1]".

Had a look at my GEDCOM source as you hinted, and found that my problem was that a couple of my Media had no titles. Simple solution for me was to add the titles in FTM and now it runs to completion ok - great plugin!

I have one remaining UDF (low priority!) - REPO.NAME.CONC which, if necessary, I can adjust in the GED file when I finally commit to moving from FTM to FH.

Cheers
John

[tatewise]

John, that is all good feedback to help Ian improve the Plugin.

As a matter of interest, did you understand the initial Plugin screen of Options?

Did you alter any of those Options or just trust the default settings?

[shoshk]

John,

Are you running 6.1?

Shosh

[JohnCulley]

Hi!

Mike: I think I understand the options, but for this first trial I used the default settings. I had it in mind to experiment with other settings when time permits ...

Shosh: I'm running v6.0.4, on 64bit W7pro SP1.

As background, I've been a long-time FTM user with currently around 650 individuals in the database, but a fair bit of peripheral info.
FTM is still my prime application, and I'm happy to wait until I'm certain I can migrate all my hard-won data with minimal (preferably no!) omissions or changes into Fh.
Although I have a working knowledge of a range of scripting languages, Lua is new to me so I'm on the steep bit of the learning curve right now!

Cheers,
John

[Barnowl]

Hi John
Thanks very much for your feedback - I will be able to fix both these problems
You and Shosh are getting the same error for different reasons.
You have no Title. In v6.1 the error I was looking for is fixed and I can't find a title in the 'bad' place.

Repository Name will be fixed too

Please if anybody finds any UDFs after running the plugin I would be very grateful for details.
And obviously any errors.

I will have a new version very soon which will handle more UDFs, including Height, Weight, Phone, Email, Separation, Medical, Excommunication, and it will output a log file. Sorry Shosh not v6.1 at this stage, but that will be the next priority.

Thanks again
Ian

[shoshk]

Ian,

I will wait patiently for your update for v6.1.

Looking forward to it...

Shosh

[Candy1837]

I have just joined the FHUG as a means to find out about changing from FTM to FH. I have used FTM since version 2 and have research from 30 years of work. I have downloaded the basic version and imported one of my files as a test. It is missing all these bits and more. It does not recognise AKA (Also Known As), it misses it altogether. I have 9 files, all various lines with over 7000 individuals most with sources. I do not want to create an immense amount of work to transfer this data as we are planning a UK trip and need to make sure I have all the data correct before heading off. From what I have read here, there is a plug in however it only works once. So what about my other 8 files? Would appreciate some feedback on this

[ColeValleyGirl]

Candy, the plugin will need to be run once *for each file* so you can apply it to all of your 9 files. I'll leave others to comment on the detail as I'm not familiar with FTM.

Helen

[tatewise]

Candy, there are several developments in progress to help FTM users migrate to FH V6 smoothly.
Both the FH developer Calico Pie, and FHUG members are working on solutions.
But Ancestry did not give much notice about the demise of FTM.

Your observation about certain data appearing to be missing is quite usual.
It almost certainly exists, but is hidden because it is invalid Gedcom produced by FTM.
The Plugins we are working on should reveal that data eventually.

I would advise you wait a while before committing to migrate to FH V6.
You have at least a year before you have to jump ship.
Why not stick with FTM for your UK trip?

[DarrylGale]

Candy - I'm a FTM user and I've done a number of test GEDCOMs into FH and most of the data is there, hidden in some cases, a few issues still to be sorted, but as Mike says Ancestry gave very little notice of the retirement of FTM and it's the FTM GEDCOM that's causing the problems - and you will find similar problems with all the other major genealogy programs!
My solution - keep using FTM, clean up your FTM database, read the FHUG forum daily, experiment on FH if something new crops up on a thread as it's a great way to learn and understand FH, and wait patiently as Calico Pie & FHUG members work together to arrive at a solution for FTM owners.

[Candy1837]

Thank you everyone and especially Mike. I am starting to get panicky about it all. I don't go to UK until Q2 2017. Do you think that I should wait until we return in June 2017? It would certainly be easier for me.

HAPPY AUSTRALIA DAY from me downunder

[tatewise]

I am confident FH will have got satisfactory importing of FTM sorted out way before 2017!

I would take bets on Q2 2016.