* Migration from PAF to FH5

[richardkendell]

I have posted an updated Plugin at plugins:wip:import_from_paf|> Import From PAF.

It would be helpful to have a sample PAF GEDCOM that you could post in the KB page above.
(I am happy to give instructions once you have KB access approved by Jane.)

On looking through earlier postings in this thread one question arises.
In the post from Richard on Mon Mar 03, 2014 8:48 am the following snippet of GEDCOM is shown.
1 EVEN
2 TYPE MARR
which suggests that PAF does not use the conventional GEDCOM 'Family as Spouse' structure for Marriage and Divorce Facts, but uses custom Individual MARR and Individual DIV Facts.
Can you throw any light on this?

Actually this Family Facts feature needs further investigation, because the concept of MARRIAGE: labels is ambiguous if there is more than one 'Family as Spouse' entry for the Individual.
Does the 1st MARRIAGE: label refer to the 1st 'Family as Spouse' entry, or does it refer to the 1st Marriage Fact in any 'Family as Spouse' entry, and similarly for DIVORCE: labels?

Mike
I am sorry I cannot comment sensibly on this, my knowledge of the inner workings of PAF are limited, perhaps John has a better idea than I have.
Richard

[Taxin]

Good morning all from a warm but cloudy Brisbane this Saturday morning

Mike, congratulations on the plugin... Last night I made a copy of my project then downloaded and 'installed' the plugin.

When I ran the plugin it identified a whole bunch of incorrect tags. These were predominantly pieces of text that I had cut-n-pasted from newspaper articles so I used the GenPad editor to modify the offending items, eg changing MEARS: The Relatives and Friends... to MEARS-The Relatives and Friends... etc. It took me several attempts but I finally got a clean run.

Then I noticed that christening events were not appearing on the individual details section of the screen (my screen is split into three left to right - a 'family' view, an 'individual' view, and on the right sources attached to the individual). So I Customised the individual by including Baptism... I exited from the program then went back in again expecting to see my baptism/christening entries but they were not showing. Back to GedPad again and I changed all PAFs "1 CHR" tags to "1 BAPM" and all christenings now appear. Woohoo.

A bit more digging around and I see that Text From Source appears in two places. One appears on the right third of my screen in the 'sources' pane and the other is when the Source entry is created. I don't understand why both fields have the same field name but that's my limited knowledge of FH5. Anyway, the plugin had used the Text From Source field of the source item (as requested) but this is not the field visible on the third part of the screen. Dang, Damn...

My PAF gedcom is 18.7Mb (just over 40,000 individuals) so later today I will trim it down a bit - just one family - and try to get the trimmed file uploaded for you to experiment with if you wish.

John

[Taxin]

Mike

You queried PAFs treatment of a divorce.

On the marriage screen there is a Divorced tick box which gives the following gedcom.
0 @F6@ FAM
1 _UID 7B8C87D8CCDBD411BD36000000000000E57A
1 DIV Y
1 HUSB @I980@
1 WIFE @I5@
1 CHIL @I10222@
1 MARR
2 DATE 26 Apr xxxx
2 PLAC xxxx

And when entering notes for an individual or a marriage it is possible to select the PAF label DIVORCE (or DIVORCE FILING - I have not used this one). Thus DIVORCE: will appear in the PAF notes we have been discussing.

I am unfamiliar with (and haven't seen) the 1 EVEN / 2 TYPE MARR tags that you mentioned.

John

[tatewise]

Good-day John it is bright and sunny here in the UK.

To change text there is a Plugin Search and Replace that is probably easier than GedPad.

In the Plugin the Baptism/Christening PAF Fact Label and Data Ref associations were:
BAPTISM: to BAPM.NOTE2 that I suspect you changed to BAPM.SOUR>TEXT
CHRISTENING: to CHR.NOTE2 that maybe you changed to CHR.SOUR>TEXT

When a PAF Label is detected then an existing Baptism/Christening Event should have been used, or a new one created.
The PAF Note text should move to the Source Record > Text From Source field, and you must open the Source Record to see it.

However, if you used Data Refs BAPM.SOUR.DATA.TEXT and CHR.SOUR.DATA.TEXT the PAF Note text should move to the Citation > Text From Source, but if the Event has no Source Record, then it will only be visible on the All tab.

The two Text From Source fields confuse many people, but are derived from the GEDCOM structure.
Source Record > Text From Source is for when you have a single document per Source Record and holds the document transcript, e.g Birth Certificate.
Citation > Text From Source is for when you have a whole register per Source Record such as a Church Parish Register, and each Citation then holds a single document transcript.

Yes, a representative PAF GEDCOM would be useful, especially if it also contains _UID and _AKA and Date Phrases.

Your example of a Family structure is just like the GEDCOM standard, although the Y after DIV is unusual.
Richard does not appear to know how the 1 EVEN / 2 TYPE MARR got created.

So one complication I predict is when an Individual has two Families/Spouses, but in the 1st Family the Spouse dies and in the 2nd Family they are Divorced.
As the Plugin stands, a single DIVORCED: Label will move its Note Text to a new Divorced Event in the 1st Family, and ignore the 2nd Family Divorce.
How does PAF deal with this dilemma?

A revision of the Plugin should be ready in a day or two, with a few extra bells & whistles, plus a User Dialogue to edit the table of PAF Labels and FH Data Refs.

[richardkendell]

Mike
I have just run your plug in and it seems to have worked excellently. It is clear I have much to do to clean up my data but I shall just have to nibble away at it.
Many thanks for the work you have put in.
Richard

[Taxin]

Good afternoon from a damp Brisbane this Sunday afternoon

Despite the damp conditions it has been a partially successful afternoon. I spent some time checking the operation of the plugin and understanding finer points of FH operation and data storage. I'm still happy with how everything has gone.

I printed out the plugin and will take that to work tomorrow for some lunch time reading.

Mike, the PAF Help file contains a passage that says for unmarried couples to use the phrase/words "Not married" in the date field of the marriage. This results in the following being saved in the gedcom file.
0 @F5@ FAM
1 _UID 798C87D8CCDBD411BD36000000000000E35A
1 HUSB @I978@
1 WIFE @I4@
1 CHIL @I979@
1 CHIL @I12665@
1 CHIL @I31454@
1 MARR
2 DATE NOT MARRIED
This date string was subsequently displayed in FH as "NOT MARRIED". Could the plugin be trained to ignore/remove this.

John

[tatewise]

The Plugin can do better than that, because it can set the GEDCOM/FH Marriage Status to Unmarried Couple or Never Married (perhaps chosen by an option).

I am looking at some enhancements to simplify the handling of the User Defined PAF Labels.
The Plugin would automatically search the Individual Notes looking for possible PAF Labels.
It already has a default PAF Labels list, but would adjust that list depending on what it found.

Are these PAF Labels normally considered to be capitalised words such as Biography, Hearth Tax, Military Service, Voting Record, and Protestation Return that only get de-spaced and CAPITALISED in the GEDCOM??

If so, the Plugin could benefit from that convention and automatically use FH Custom Fact Names that matched.
(The user may have to put spaces & Capitals in some PAF Labels listed by the Plugin, but that is a trivial task.)
In any case the Change Any Fact Tag Plugin could correct any mistakes afterwards, before the Custom Facts get fully defined by the user.

(BTW: Thanks John for the sample GEDCOM.)

[Taxin]

Good evening all

Mike, re your suggestion of Unmarried Couple or Never Married. I did a quick bit of research and think I will select the Unmarried Couple option for my records.

It seems to me that a PAF note might be a little more complex than first thought. It seems that a PAF note has a structure similar to:
<PAF label><some text>
<some more text>
blank line
<PAF label><some text>
<some more text>
The blank line does not appear - unless inserted by the user - when there is only PAF label but seems to be automatically inserted immediately before each subsequent PAF label is inserted into the Note field. With a structure like my sample the PAF program will 'see' the presence of the two PAF labels, but if the blank line is removed the PAF program will only see the first of the labels. It is as if the program is assuming a PAF block of text ends with the following blank line, eg two newline characters in sequence. The blank line is not part of the user's text.

John

[tatewise]

The Unmarried Couple versus Never Married choice could be a Plugin tick box option.

The blank line before PAF Labels in fact will help the Plugin distinguish Labels from lines that happen to start with upper-case letters and a colon.

[tatewise]

V1.2 with a user dialogue, plus options for _UID and _AKA and Date Phrases is available via the Work in Progress page.

Please let me have your feedback, especially when run on a newly imported PAF GEDCOM.

[tatewise]

I have posted V1.3 of the Plugin at plugins:wip:import_from_paf|> Import From PAF.

Could John and Richard, and anyone else interested, please give it a try.

I think it offers virtually all the options requested so far, and mostly by simple drop-down list selection.

The PAF Labels are automatically detected, and default destination text fields offered, but can easily be altered.

In addition the _UID and _AKA fields can be deleted, or moved to various destinations.

Also most Date Phrases can be converted to valid Dates.

Checkout the Plugin Help and Advice for further information.

If I get positive feedback then it can be published to the Plugin Store.

[Taxin]

Good morning/evening all

First, my apologies for the short absence from the group but I had a computer problem that had to be sorted.

Mike, I downloaded the plugin then 'installed' it - no problem. When I selected 'Run' the screen went quiet for a few seconds then a dialog box was displayed. Problem is, the top of the dialog box disappears out the top of my physical screen - I see only the lower part of the red text - and the bottom of the dialog box disappears past the bottom of my physical screen - I see only the upper part of the red text, and do not see any buttons. Can this dialog be given vertical scroll bars?

Apart from that, I am impressed with the content of the dialog box.

John

[tatewise]

I have put V1.4 in the WiP page, although it does NOT have a scroll bar, the four buttons that were at the bottom are now at the top, so they should be accessible.

I presume the problem is caused by a large number of PAF Notes Labels (how many?) together with a relatively small screen and relatively large fonts.

To reduce the Plugin window size try using Set Window Fonts button at the top, and set the Body Text Font Size to a smaller number. You can often type in a smaller number if the drop list has no smaller numbers. Finally click the Minimum Size button.

[Taxin]

Good morning all

Having re-established communications with my computer again I spent a few hours yesterday investigating the converted gedcom.

Overall, I like what I see. I do have a request, which if it can be accommodated, would round out this plugin but I'm not sure that it is feasible.

Imagine some text:

This is a paragraph from a newspaper which continues for several lines, and when this is entered into PAF we see a mixture of CONC and CONT lines.
<-- this blank line is important
And this is another paragraph from the same newspaper article.

When we inspect the PAF gedcom we see a NOTE that looks something like this:

1 NOTE NEWSPAPER: This is a paragraph from a newspaper which continues for several lines,
2 CONC and when this is entered into PAF we see a mixture
2 CONT of CONC and CONT lines
2 CONT
2 CONT NEWSPAPER: And this is another paragraph from the same
2 CONT newspaper article.

The plugin currently creates two NEWSPAPER facts because it does not know, and has no way of determining, that the second label is a continuation of the first label.

Could we make an assumption that sequential instances of the same label can be aggregated into one FH fact? It would work in the above example but would it be correct in the general sense. All thoughts appreciated.

John

[tatewise]

I feel uncomfortable with that concept, because for other Facts it is less likely that just because they are sequential that they should be aggregated, e.g. Occupation, Education, Census, Marriage, Address, and even Biography.

Is it a significant problem? Are there a lot of such cases?

If only a few then either the Notes could be edited beforehand to remove unwanted NEWSPAPER Labels, or the Facts merged afterwards.

It can be resolved by using the Plugin Search and Replace but it is quite a complex series of commands to get the desired effect without any side-effects.
However, it could be used to Search and Replace every Note where multiple NEWSPAPER Labels occur, and interactively decide which are sequential and just edit those.
The settings would be LUA Pattern Mode and Note Fields Individual (INDI.NOTE).
Search:
(
NEWSPAPER: .-

)NEWSPAPER:

Replace:
%1

But every case would need reviewing and sometimes manual editing to get the correct effect.

Regarding the Plugin:
If it is only likely to apply to NEWSPAPER: Labels then the easiest solution is to have two Fact Options:
Newspaper (distinct)
Newspaper (merged)

Otherwise a Merge tick column could added as an option for every Label.

[Taxin]

Hi Mike,

Thanks for the prompt reply, and it was along the lines of what I expected.

I used the PAF label NEWSPAPER but this issue could apply to any of the labels used within the PAF notes. I can't speak for others but I have more than a few instances. Rather than tie up a scarce resource (yourself) I'll begin a background task of manually identifying and fixing each instance. That will also give me a chance to correct any other issues that I see.

John

[tatewise]

I'm ahead of you, try Plugin V1.5 from the WiP page.

Could you also please review the how_to:import_from_paf|> Import from Personal Ancestral File (PAF) page, and the Plugin Help and Advice pages also found at plugins:help:import_from_paf:import_from_paf|> Import From PAF Plugin.

Another scenario that needs clarifying is this:-

Consider that there may be a number of PAF Labels for say BIRTH, CENSUS, DEATH, etc.
There may be a number of existing Facts for the same Individual, perhaps Birth, Census, Death, etc.

I have assumed so far that the text against the 1st BIRTH Label is moved to the 1st existing Birth Fact, the text against the 1st CENSUS Label is moved to the 1st existing Census Fact, and so on...
New Facts are created only when no existing matching Fact is available.

With this assumption, should the Plugin raise a warning message if there is text against a Label for which no matching Fact already exists?

Alternatively, maybe the text against Labels should always be creating NEW Facts for the text rather using existing Facts?

It all rather depends on how PAF works. What say you?

[Taxin]

Good morning Mike

In the Import from Personal Ancestral File (PAF) page you mention upgrading to the latest version of PAF. The latest version of PAF, v5.2 from memory, was retired in 2013. While it is no longer available from the FamilySearch.org web site it may still be found on some other sites.

In the Import From PAF Plugin page, under the first dot point, could you mention the differences between Apart, Join and Merge values in the Mode column.

Re your comment on the warning message. When I first read your comment and thought about my data - just over 40,000 individuals - I pictured a rather large warning report, but after a bit of quiet reflection I think the warning message is a good idea and it will serve to highlight to the user those individuals requiring particular attention. The report can always be printed then used as a 'to do' check list.

Over the course of today I have run the plugin quite a few times. With my data I get the best 'conversion' when I set Marriage mode = Join (no change), Census and ElectoralRoll mode = Apart (because each of these must have a separate year/date attribute and some of mine do not), and mode of other PAF labels = Merge. I will do some more testing tomorrow.

John

[tatewise]

how_to:import_from_paf|> Import from Personal Ancestral File (PAF)
Do you, or anyone else, happen to have a PAF 5.2 download that we could add to the FHUG Downloads?

plugins:help:import_from_paf:import_from_paf|> Import From PAF Plugin
On the initial summary page, none of the bullet points mention details of any user options.
The User Options page describes the options, and the Note Labels Features page expands the PAF Label Options further including the Mode option.
Did you follow those links, and also check the FAQ page?

Is my assumption correct that text against the 1st BIRTH Label should move to the 1st existing Birth Fact, etc, etc.

Could you explain why Marriage mode = Join works best for you, and what (no change) means?

If Merge works best for most other cases, does that mean my assumption above is incorrect?

Ideally, I would prefer the default Mode to work for the majority of users especially if they only use the default PAF Labels.

[PeterR]

Do you, or anyone else, happen to have a PAF 5.2 download that we could add to the FHUG Downloads?

I have a copy of PAF5EnglishSetup.exe at https://db.tt/Ojr6qWyt (Product version 5.2.18.0).

[tatewise]

Thanks Peter ~ that has been added to how_to:import_from_paf|> Import from Personal Ancestral File (PAF) page.

[tatewise]

Now having a working version of PAF 5.2 to play with, I have a better understanding of how Note Tags function.
I see now that in the Plugin, I should rename Note Labels to Note Tags.

One part of the Help says:

The tag is one word that is typed at the beginning of the note in all uppercase letters, followed by a colon:
If you want to use tag with more than one word, type a hyphen instead of a space between the words.
Note: An underscore does not separate words in a tag.

But elsewhere says:

Tip: In the Notes Selector screen, a tag can have only one word. If a tag has more than one word, type a hyphen or underscore instead of a space.

However, the Notes Selector screen allows and detects Tags comprising virtually any characters!!!!
And clearly many users create Tags with digits.

Perhaps my Plugin should automatically replace each Tag hyphen and underscore with a space character for creating the Custom Fact name.
Also my Plugin should be more tolerant of the range of characters possible in Tags.

When the Notes Selector screen is used to select and Open any particular Tag, then if a blank line is inserted, an extra Tag label is automatically inserted.
Only the Open All option allows the same tag to be inserted non-consecutively.
So the default Mode should be to Join consecutive tag labels.

I notice that it is difficult for a Tag to be accidentally inserted in a Note because the Notes Selector screen detects it and lists it * GREYED_OUT.

There appears to be no association at all between Note Tags and Facts, so perhaps my Plugin should always create new Facts for each Note Tag rather than add the text to existing Facts???

[Taxin]

Mike, good morning from an overcast Brisbane this Sunday morning.

I'm comfortable with your use of the term Note Labels or Note Tags as long as we are consistent in our usage, which PAF was not. Coming from an IT background I prefer to use underscore or camel case rather than hyphens when using multi-word names, eg. Electoral_Roll or ElectoralRoll rather than Electoral-Roll or Electoral<space>Roll.

During my time in using PAF I created a few non-standard Note Tags and as you have observed these are displayed in light grey. Invariably my tags were all typing errors so I was able to correct these.

You are correct is saying that PAF has no explicit relationship between an individual's facts and named Note Tags. I suspect that many PAF users would have stayed with the default set of PAF Events and would have used their Notes to capture other facts. I would agree that if a PAF user has created a named tag then that is the type of fact they are trying to capture and hence a new FH fact would be appropriate. Would you recommend the use of Mode = Apart in these cases?

John

[tatewise]

Good morning John from sunny Torbay.

I am really looking for advice regarding typical PAF usage rather your personal preferences.

The complications with mapping PAF Note Tags to FH Fact Names in general are as follows:

• A PAF Note Tag cannot use spaces, and - or _ are advised instead, so would PAF users expect those to become spaces in a FH Fact Name???
• A PAF Note Tag can actually use most characters including upper & lower case letters, but most PAF users appear to stick to upper-case & digits only, and the Plugin converts to lower-case except for 1st letter for the FH Fact Name, but this will result in shared Fact Names for PAF Note Tags such as ElectoralRoll and ELECTORALROLL. Is this likely to be a problem???
• Any FH Fact Name can use most characters, but the mapping to internal Fact Tag converts all letters to upper-case, converts space to _, retains digits, and discards all other symbols including - and _. Thus PAF Note Tags such as Electoral_Roll, Electoral-Roll, ElectoralRoll, and ELECTORALROLL would all share the same Fact Name. Is this likely to be a problem???

I may have insufficiently explained the problems with moving PAF Note Tag text to existing Facts/Events.
There is no problem if all the PAF Note Tags are different from all Personal/Marriage Events recorded in the data.
The problem arises if there are PAF Note Tags such as BIRTH & DEATH as well as Personal Events such as Birth & Death for the same person.
Should the PAF Note Tag text for say BIRTH be moved to the pre-existing Birth Event or should a second Birth Event be created for the text???

You say PAF users would use default PAF Events, and only use Note Tags for other things, but the default Note Tags are almost the same as the commonest Events so the possibility for a clash is increased.

With this in mind the Mode options may need extending, because currently they are Tag focused not Fact/Event focused.
The way PAF handles Note Tags often leads to consecutive identical Tags, so the default Mode should probably be Join to collapse consecutive identical Tag text into one instance.
Apart was meant to be the opposite and keep consecutive identical Tag text apart.
Merge is a more aggressive form of Join in which all identical Tag text is merged into one instance.

None of those identify whether the text should be appended to an existing Event in the data, or create a new instance of the Event.
That would need say an Old or New suffix for each Mode to say whether to use Old or create New data Events for the text.

[richardkendell]

Mike
I have just seen your message of 19 March having returned from holiday.
I used the first version you wrote and it did everything I wanted/needed. All I have to do now is tidy my data up and a big (but worthwhile) task it is proving to be.
Thank you for the work you have put in.
Richard