* UNCAT UDFs

[Robert Jacobs]

After my conversion from RootsMagic I have manged to get rid of most of the UDF fields using the Change Any Fact plugin. I am left with about 1300 UNCAT UDFs, most of which have come from the RM fact description field. I would like to change them all to notes, but I cannot find the UNCATs in the Change Any Fact plug-in or in the Tools/Fact type. How can I recover this data?

[tatewise]

I presume you have been following the advice in Handling Uncategorised Data Fields.
Have you got a comprehensive Result Set from the UDF List plugin?

The advice in Importing to Family Historian for RootsMagic suggest that many UDF will be such as _UID and _TYPE and _SCBK and _PRIM that can be deleted.

The Change Any Fact Tag plugin only operates on the Fact level 1 GEDCOM tags.
Which also explains why Tools > Fact Types does not list those UDF.

The only automated way to change those UDF is with a custom plugin.
That will need precise details of exactly what structure the UDF take and exactly where you want then to end up.
A screenshot of some samples would help as I suspect the GEDCOM tag is not UNCAT but perhaps _UNCAT.

[Robert Jacobs]

Thank you, Mike. It is indeed _UNCAT. I've attached a small file showing what a typical one looks like.

The best disposition of these would be to have them attached to the underlying fact as a note; that way I can work them into appropriate sentences.

Thanks so much for your help.

[tatewise]

If that is a typical example then it is somewhat different to what you described.

If that were a standard GEDCOM Graduation attribute then a degree in dentistry would be its value/description.
In the Facts tab it would be shown in the box against the label Graduation: just above the Date: and Age: boxes.
Then that attribute value could be incorporated into the global Sentence Template using the {value} code.

The first thing to establish is whether that is a standard Graduation attribute by showing its appearance on the Facts tab including the Sentence box at the bottom.

If all your _UNCAT values are similarly associated with standard attributes then they can all be handled the same way with a very simple plugin based on one I already have that simply moves the _UNCAT value to the Attribute value.

If what you imported from RM was a standard Graduation attribute then its value/description would not have moved to _UNCAT, so there is some history to this that would be useful to know and may reveal a better solution.

[Robert Jacobs]

I believe they are all associated with standard values; they all comprised description fields for facts that FH7 has already accepted. I would very much like to try your plug-in.

Mea culpa, however. On reviewing my file loading preference I noticed that I failed to tick "Move invalkid data into note fields where possible. I could redo the entire conversion with the field ticked; do you think that might put the _UNCATs where I need them to be?

Again my thanks.

[Robert Jacobs]

I meant attributes, not "values"

[tatewise]

I need to sign off now and we can continue tomorrow.

If they really were genuine standard Graduation attributes then their description/value would not have been moved anywhere as it would not have been invalid data. I suspect something else is happening.
Can you post a fragment of GEDCOM exported from RM for that Graduation attribute before import to FH?

[Robert Jacobs]

Dear Mike,

I think I may have this aspect of my conversion solved. I started over after ticking the box for "Move invalid data into note fields where possible." It looks as though _UNCAT has completely disappeared from the list of UDFs. I believe that the only remaining UDFs are the RM7 sort dates: _SDATE. I'd used them extensively in RootsMagic to properly place married names. They seem not to interfere with anything. Unless you advise otherwise I think I'll just leave them alone.

For your information, here is the GEDCOM fragment from which the original _UNCAT came:

1 EDUC biology

2 _SDATE 17 NOV 1943
2 PLAC Seattle, King County, Washington
2 ADDR the University of Washington
2 SOUR @S46@
1 GRAD a degree in dentistry

2 DATE 1948
2 PLAC Chicago, Cook County, Illinois
2 ADDR Northwestern University Dental School
2 SOUR @S81@
3 PAGE Seattle <i>Times</i> obituary, 4/27/2005

I can't thank you enough for all your help.

Robert

[tatewise]

Let me deal with the two issues of _SDATE and _UNCAT below.

_SDATE
This is NOT usually a UDF but is a perfectly valid FH fact Sort Date just like in RM.
Maybe you noticed that UDF tags often have an underscore prefix and assumed that all tags with an underscore are UDF.
That is a bit like saying because all cats have four legs then everything with four legs is a cat.
The underscore prefix indicates the tag is an extension to the GEDCOM standard, which is allowed by the standard, but may not be recognized by other products. See the GEDCOM Extension List for FH V6 which does not yet include all the extensions for FH V7 such as _SDATE.

If you run the UDF List plugin it will NOT list _SDATE as a UDF unless it is used in an invalid way.

So let's review your example EDUCation attribute with _SDATE 17 Nov 1943
Display that Education fact in the Facts tab of the Property Box and its date will have (s) and both the actual and sort Dates will be shown below.
Switch to the All tab and fully expand the nested details. The Sort Date does NOT have a star bullet, so it is NOT a UDF.

_UNCAT
This is a bit more mysterious as you now cannot recreate them with the GEDCOM you illustrated yesterday.

The GEDCOM specification does NOT allow description values on events such as Birth, Marriage, Census, Death, Burial...
Whereas it does allow description values on attribute facts such as Occupation, Education, Graduation, Possessions...
However, many products such as RM do not enforce that rule, and FH has a problem when importing their GEDCOM.
FH will move such event description values to either the local Note or the _UNCAT field.
That is chosen by Tools > Preferences > File Load/Save option Move invalid data into note fields where possible.

So in the past, maybe that Graduation fact was actually a custom event in RM which when imported into FH would move its description value to _UNCAT and you subsequently used Change Any Fact Tag to convert the custom event to the standard GRADuation attribute.

Is it also possible that in RM you have similarly changed that and other custom events to a standard attribute, so that now the exported GEDCOM does not pose a problem for FH?

Can you throw any further light on this mystery?

[Robert Jacobs]

Dear Mike,

The _SDATE example I sent does indeed convert to a legitimate FH7 sort date. The UDF list generated by the plugin, however, shows hundreds of instances of sort dates which do show the star bullet. Most of them, are RM7 sort dates which were connected to the ALT NAME fact in RM7 — mostly married names, but also some changed immigration names.

The UDF list also shows some entries related to MAP, LATI, LONG and about a dozen that look like this:

"28 AUG 1959" "Ann Masha SAMOLS" "INDI" "DATE" "INDI.NAME.DATE"

which completely puzzle me.

As to _UNCAT, I reconverted my RM7 file after ticking "Move invalid data into note fields where possible." The date was put into the appropriate note fields and can be used with minor tweaks to the sentence templates. I still have the conversion with the _UNCAT fields; I could run a test of some kind if it would be useful to you.

"Graduation" is a built-in RM7 fact tag, so not a custom event. I may have changed the RM7 sentence template, but that would have been the only change.

Robert

I attached the UDF list in case it might be useful for you.

[tatewise]

UDF
Than you for the UDF List which I explain below.
UDF can arise for two reasons. Either the tag is completely unrecognised, or it is used in an invalid context.

Most of your UDF are _SDATE and DATE tags subsidiary to an INDIvidual record NAME tag.
NAME is NOT a Fact in strict GEDCOM terms, so cannot have Date fields if any type, and thus they are invalid.

Most of the remainder are related to invalid subsidiary tags on an ADDRess tag.
MAP & LATI & LONG are only allowed in strict GEDCOM terms on PLACe fields.
NOTE fields are not allowed on ADDRess fields anywhere.

_UNCAT
I am puzzled by the GRADuation fact with _UNCAT data.
For this analysis, I would prefer you did NOT tick Move invalid data into note fields where possible as once moved into a Note field the description/value cannot be distinguished from other Note text, so cannot be used in Sentence Templates.
Whereas it is uniquely identifiable when moved into the _UNCAT fields which are easily found as UDF.
The conditions under which those two alternatives occur are identical, and I need to discover what those conditions are.

The conditions should be that the imported fact is an Event and has a description value on the same line in the GEDCOM.

What you posted earlier was 1 GRAD a degree in dentistry
That is a perfectly valid Attribute that is allowed to have a description value on the same line.
I cannot reconcile that with your even earlier screenshot showing a Graduation fact with _UNCAT a degree in dentistry and matching Date, Place, Address fields, which cannot happen unless that Graduation fact was somehow being considered to be an Event when imported to FH.

Have you a GEDCOM file containing that 1 GRAD a degree in dentistry which when imported to FH produces the _UNCAT a degree in dentistry example you posted earlier?
If so, then please post that entire fragment of GEDCOM for the same 1 GRAD fact.

[Robert Jacobs]

Dear Mike,

Here is the fragment. Move data into note fields was unticked. It results in "a degree in dentistry" showing us as _UNCAT with a star bullet. I should add that in the original conversion which I sent you I had edited out all the "2 _PRIM Y" entries from the source GEDCOM. As you see, I didn't bother to do that this time.

1 RESI
2 _PRIM Y
2 DATE 11 APR 1930
2 PLAC Lynden, Whatcom County, Washington
2 SOUR @S20@
1 EDUC biology
2 _PRIM Y
2 _SDATE 17 NOV 1943
2 PLAC Seattle, King County, Washington
2 ADDR the University of Washington
2 SOUR @S46@
1 GRAD a degree in dentistry
2 _PRIM Y
2 DATE 1948
2 PLAC Chicago, Cook County, Illinois
2 ADDR Northwestern University Dental School
2 SOUR @S81@
3 PAGE Seattle <i>Times</i> obituary, 4/27/2005
1 OCCU dentist
2 _PRIM Y
2 DATE FROM 1949 TO 2003
2 PLAC Seattle, King County, Washington
2 SOUR @S46@
1 EVEN as a dentist in the Navy during the Korean war,
2 TYPE Military
2 _PRIM Y
2 DATE CA 1951
2 SOUR @S81@
3 PAGE Seattle <i>Times</i> obituary, 4/27/2005

I hope this helps and that we'll find a way to resolve all these UDFs. Thanks again for your help.

Robert

[tatewise]

I have to apologise for inconveniencing you. I was convinced that GRADuation was an Attribute like EDUCation, but after checking, I find it is actually a standard Event That explains why a degree in dentistry is moved to _UNCAT (or a local Note).

All the other facts in your example retain their description value as expected.

Regarding UDF, there is not much that can be done with the _SDATE, DATE, MAP, etc, cases I explained earlier.
There is no way of handling the _SDATE and DATE on NAME fields unless the NAME tags can be converted into Facts.
It is unlikely that the ADDRess MAP LATI/LONG values can be moved to an associated Place record.
The remainder are just invalid.
So they will all probably have to be deleted.

That just leaves the _UNCAT cases. Do you have many of those?

[Robert Jacobs]

Dear Mike,

I have several hundred _UNCAT cases. Is there any reason I shouldn't avoid them by ticking "Move invalid data into note fields where possible?" That will preserve the data, and in most cases I can fix the resulting reports by changing the default FH7 default sentences.

As for the others, I'm perfectly happy to delete them all but I don't know how. They don't show up in either the change any fact plug-in or the /tools/fact type tool. I only see them on the report generated by the UDF list plugin.

Robert

[tatewise]

The article Handling Uncategorised Data Fields explains how to delete them from the UDF List Result Set.

As I said before, once moved into a Note field the description/value cannot be distinguished from other Note text, so cannot be reliably used in Sentence Templates. Whereas it is uniquely identifiable when moved into the _UNCAT fields. The best way forward is to import into _UNCAT fields and then use a Plugin to move the text to a labelled Note, which can be identified in Sentence Templates. See Narrative Report Fact Sentence Templates under Custom Fact Fields to understand how that works. I have Plugins that move _UNCAT fields in various ways that could be adapted to work for labelled Notes.

[Robert Jacobs]

Thank you, Mike. I'll do a fresh conversion tonight and clean up the various UID entries. I also seem to have some Family Search codes. Is there any point in keeping them?

Your _UNCAT plugins are sure to be helpful, and I'm still hoping to learn how to deal with _SDATE.

Thanks again for all your help. I've just realized that it must be getting close to the end of the day where you are. I hope we can resume tomorrow

Robert

[tatewise]

I am unsure what you mean by Family Search codes and only you can decide if they are useful.
Exactly what do they look like? Are they data attached to a UDF? Details please.

What do you mean by hoping to learn how to deal with _SDATE?
The Sort Dates associated with Facts are OK.
The _SDATE UDF are associated with Names, so how are you hoping to deal with them?

If the _UNCAT Plugin moved the text into the Fact local NOTE using the following format would that be OK?
[[
Value: <Text from _UNCAT here>
]]

It would be easy using a simple Search and Replace to alter the label from Value: to something else either globally or on a Fact by Fact basis.

[Robert Jacobs]

Dear Mike,

I don't have an _UNCAT plugin and do not see it on the plugin store. The _UNCAT UDFs do not show up in the Change Any Fact plug in. The format you suggested for the _UNCATs would be perfect. I think, however, that when I convert (RM7 --> FH7.02) with "Move invalid data into note fields where possible?" I do get the _UNCATs put in local notes. For example, adding {note} to the graduation template the words "a degree in dentistry" are appended to the end of the sentence. That's from the GEDCOM fragment I sent you.

I would like to keep the sort dates that are associated with facts and get rid of those associated with names. _SDATE shows up in my UDF list but not in the Change Any Fact plugin.

The family search codes I mentioned show up as _FSFTID on the list of UDF Tags in the Change Any Fact Plugin. I that the data are the Family Search identification codes for individuals in their system, for example LKST-Y3V. The plugin allows me to discard them and that is what I shall do.

In addition to the _SDATE issue I hope there is a way to delete all the UDFs which I cannot see in the Change Any Fact plugin.

I do hope these conversion problems aren't burdening you unduly. You've been terrifically helpful already. Thanks so much.

Robert

[tatewise]

There is no _UNCAT Plugin yet. I have to write it as an adaptation of similar earlier ones I have written.
I just needed to know you would be happy with what I was proposing before I wasted time writing something unsuitable.

Certainly, if an imported Event from RM does not have an existing Note and you never modify that Note, then the only Note text will be the description value and you can use {note} in the Sentence Template. However, can you guarantee that both those conditions will always apply to every such Event?

Using a labelled Note text ensures the description value can be uniquely extracted from the Note in a Sentence Template by using the {=GetLabelledText(%FACT.NOTE2%,"Value: ")} function as explained in the article I mentioned. Did you read it?

The article Handling Uncategorised Data Fields explains how to delete any of the UDF including _SDATE from the UDF List plugin Result Set. Do I need to talk you through the process?

You could use the Change Any Fact Tag plugin to convert _FSFTID to a custom Family Search Id Attribute and retain the Family Search Id if you wish.

[Robert Jacobs]

Thank you, Mike. I would very much like to have an _UNCAT plugin along the lines you suggested.

My conversion from RootsMagic is going to be a one-time affair. The new version of RootsMagic is unusable, at least by me, and I plan to abandon it completely.

The most frequent output for my genealogical work is a website generated by John Cardinal's GedSite program. I am hoping that your Export Gedcom File plugin will enable me to continue the website. (If it interests you, it can be found at https://genarchives.com/NineGenerations/up/index.htm)

I'll re-read Handling Uncategorised Data Fields to see if I can figure out how to deal with _SDATE and the others that show up only on the UDF list plugin.

Robert

[tatewise]

Please try the attached Move _UNCAT to Labelled Note plugin Version 0.1 Date 08 Feb 2021
Ensure you have a suitable Project backup before running the plugin.
If the results are unsatisfactory then use Edit > Undo Plugin Updates before closing FH.

My Export Gedcom File plugin has an Export Mode specifically for GedSite and I'm in regular contact with John Cardinal.
It has a special feature that includes the FH Sentence Templates in the exported GEDCOM file so they can be used with GedSite if desired.

[Robert Jacobs]

Dear Mike,

The Move _UNCAT to Labelled Note plug in has successfully saved all the data from the RM7 [desc] fields and put them into the the notes section of the individual's property box. The format is [[data]]. However, I am unable to use the data in sentences either by adding {note} or {value} to the template. Could it be that the brackets are acting as some kind of escape character? Alternatively, I'm probably doing something wrong.

On another front, I'm very glad to report that the Export Gedcom plugin makes a .ged file that works well in GedSite. I'll need to make only one or two very minor tweaks to keep my website going.

Finally, I re-read Handling Uncategorised Data Fields and found that I had too hastily scanned over the crucial paragraph. I now know how to get rid of _SDATE, _LATI, _LONG etc.

Robert

[tatewise]

Robert, I have explained all that before, but I think the details are going over your head.
See Narrative Report Fact Sentence Templates under Custom Fact Fields to understand how it works, what the brackets are for, and how to use the label in Sentence Templates, etc.
The Note text format should be as below with [[privacy]] brackets and Value: label
[[
Value: <Text from _UNCAT here>
]]

Yesterday I said:

Using a labelled Note text ensures the description value can be uniquely extracted from the Note in a Sentence Template by using the {=GetLabelledText(%FACT.NOTE2%,"Value: ")} function as explained in the article I mentioned. Did you read it?

[Robert Jacobs]

Sorry, Mike. I think I've got it in focus now. Your plugin seems to work perfectly.

My remaining problem is that GedSite does not support Family Historian functions in generating sentences but does support a subset of Data References.

I have to think carefully about how best to proceed. If I use the function, I can't get the data into GedSite, but if I strip [[Value and ]] from the note I can use the data as a {note}. In RootsMagic, notes can be appended to sentences but not inserted into the middle of them. So if I globally change my FH7 templates to end with {note} I should be able to get away with a minimum of clean-up.

Thanks so much for your help.

[tatewise]

I cannot remember if GedSite supports the {cause} code in Sentence Templates but it implies it supports all such codes.
If so, then the _UNCAT description value could have been moved to the Fact Cause field which is a standard GEDCOM field for all Facts. Currently, it is only shown on the Facts tab for Death events but is available on the All tab for all Facts. A recent report to Calico Pie asked if it could be made available on the Facts tab for all Facts and the reply said "So why can't you specify 'Cause' as a required field for the Facts tab? I can't think of any good reason. We have made a note to look at that. Thank you for raising this issue." See Attempting to create a Hospital Admission fact (18765).

So don't change any of those labelled Notes just yet as you may like to use the Cause field and the label would allow a plugin to move the text.