Family Historian User Group

Yes, that makes sense on tables, thanks. I’ll tidy it up in the next version (and then iterate with pairs rather than ipairs - I’ve used that technique before, so not sure why I didn’t apply it here!).

I didn’t build in a check for blank or non-numeric TID values as I assumed (probably incorrectly) that could not happen. Let’s see what Jackie reports back, but I might have to add that so the plugin at least fails gracefully and identifies the errant source rather than crashing.

As an aside if we do go down that route - is it possible/easy to force an undo of plugin changes so far, so we don’t end up with a half-converted project?

No, I don't think it is possible to force a Plugin to Undo Updates.
The only options are:
1) Raise an error(...) which gives the user the choice to Undo Updates or not.
2) Identify the problem to the user and continue to convert as much as possible, but perhaps advise that Edit > Undo Plugin Updates is a good idea.

Thanks - #2 is what I do if it finds a problem when converting UDF fields, so I’ll carry on with that approach.

What message do you get please? Once we know what is causing the problem, we can move on to why and fix it.

Hope this helps. Keep reading after screenshot.
The editor also shows this:
Don't know if it helps, but I poked around in my RM database, and I think the only two built-in source templates I'm using are Book image copy (online) and Newspaper, Online Archives.

Thanks Jackie - it all helps build up a picture. I'm going to modify the plugin so it makes fewer assumptions about what I think the RM database should look like, and as far as possible identify problem records and data fields rather than just crashing.

Another option might be for you to do a very selective GEDCOM export from your RM database, selecting just a few (non-living) individuals who have facts supported by these built in sources. Just select "Sources" and "Extra details (RM Specific)" from the export options to give a minimalist file but one that should contain enough information to help diagnose the problem. Would that be possible please? If you are prepared to share that but don't want to post it here, send me a PM and I'll give you my e-mail address so you can send it directly.

Thanks.

do a very selective GEDCOM export from your RM database

Done and attached. These people are long gone. I did replace my personal info in the header with fake info. I ran it with the plugin and it does the same thing. Hope it helps.

Great, thanks. It will probably be tomorrow before I get back to you, as I want to make the plugin more resilient before I post another update.

Actually, it's not going to help. I thought I had the ran the plugin against the new gedcom, but I didn't. I did it again, with the gedcom I sent, and it runs the whole way through. I should have some time this afternoon to make a Gedcom with no living and see if that one errors out.

do a very selective GEDCOM export from your RM database

I narrowed it down to one person who will generate the error. I played around a bit to see if I could isolate the cause, Looked at all his sources in RM, ran all the database tools, etc. So here is his gedcom. Hope it helps.

Found it! I'll blame an "unexpected characteristic" in the RM export - I won't call it a bug, there may be good reason to do it that way . Templated data are stored as a NAME/VALUE pair for each data field, both in RM and FH. Sometimes, empty fields are exported just as a NAME with no associated value. I'm note sure why, as it does not appear to be consistent.

If that field was also missing from the template definition, we ended up with a UDF data field that the plugin didn't know how to process, and crashed. It's not clear to me how RM decides what to export. My test dataset from the web of 400+ sources appeared to come across ok, even though most of them were unused in my RM file. Certainly, in your test file there was a field that was included in individual's data when it shouldn't have been, and missing from the template definitions when it should have been included!

Yesterday's plugin update managed to process the GEDCOM without problems, as it does much more checking, and discards just bare NAMES with no VALUE before reassigning the data to the correct place. I'm optimistic that will work ok on the full dataset. I need to do just a little more checking and tidying up, but I'll post it later today.

Thanks for sticking with me - it would have been much harder to find the problem without being able to focus on your small test dataset.

Thanks for sticking with me -

I'm very grateful you are doing this. It will save me (and many others) so much time.

I've put so much error checking in the plugin that I think it is finding an inconsistency in the GEDCOM file that Jackie posted.

Lines 2410-2415 of the GEDCOM file define the field CreditLine as a Source Level field (ISDETAIL = N) for User-Defined Template 10032 (TID tag on line 2346).

Line 1197 (2 TID 10032) records that Source Record @S373@ is based on this Template.

However, when this source is cited for the birth of Raymond Russel McDonald, the CreditLine is stored at Citation level, not Source level (lines 401-404).

FH imports faithfully what is in the GEDCOM. The plugin will flag this as a problem when moving UDF data back into their proper place, and leave this field untouched. UDF fields are only deleted after moving is confirmed as successful.

Can I have a second opinion please? Is it a GEDCOM error? I've no way of knowing if this is a one-off possibly caused by a very selected GEDCOM export, or whether such errors will be common. We'll find out soon enough!

Your analysis looks correct.
Did you notice that Source record @S373@ also has a CreditLine field but with no value.
So that field appears at both the Source and Citation level in association with TID 10032.

Maybe Jackie could review template 10032 !Ancestry Basic Images Template
How does it define the CreditLine field?

Also perhaps she could review that Source record and the Citation.

It's Source level in the standard template that Jackie presumably based hers on...

Getting my morning started but saw this. I'll take a look in about an hour and post.

Wanted to give you what I have so far, but I'm continuing to look at a copy of my RM database. I've played around with SQL Lite and am going to look at the tables and see if I see anything odd. I've also tried greatly simplifying that template and so far haven't found anything that helped. I also tried putting something in the CreditLine and that didn't help.

Meanwhile, this is the template:
And this is how it looks in the citation:

Curious - the GEDCOM says the field is occupied, but the original is blank. Is the RM export corrupting records as well...?

4 FIELD
5 NAME CreditLine
5 VALUE citing Pennsylvania Historical and Museum Commission, Harrisburg, Pennsylvania, Pennsylvania County Marriages, 1852-1973; County: Fulton; Year Range: 1947 - 1956; Roll Number: 549832

Meanwhile, have a try with the latest plugin update. Does it complete without crashing? Are there any data errors reported?

(Attachment deleted).

It still crashes, with the error below. I may have noticed something else though. Let me make sure it has something to do with this. It involves 2 sources made from the !Ancestry Basic Images Template, name very similarly, and one source has the credit line filled, the other does not. It shouldn't matter but I think possibly RM is confusing the two on export. But I need to look at the full Gedcom some more. Meanwhile here is the error:

Jackie, that is not a crash. It is one of Mark's new diagnostic messages.
Crashes say Plugin Error with a red cross. These messages say Family Historian and have a warning triangle.

Mark, the blank CreditLine in the upper yellow Source record pane exactly matches the GEDCOM for @S373@ that also has a blank CreditLine entry:
2 FIELD
3 NAME CreditLine

It is the lower green Source Details pane that omits the CreditLine Citation for the birth of Raymond Russell McDonald included in the exported GEDCOM:
4 FIELD
5 NAME CreditLine
5 VALUE citing Pennsylvania Historical and Museum Commission, Harrisburg, Pennsylvania, Pennsylvania County Marriages, 1852-1973; County: Fulton; Year Range: 1947 - 1956; Roll Number: 549832

So the GEDCOM is including data not shown in the RM7 GUI. I wonder if that is something left over from some earlier variant of the Source Template. It is odd that the Source record [CreditLine] definition sits amongst Citation definitions, as if it was itself once a Citation definition.

That should be just flagging the inconsistency discussed above. What happens when you click on OK to dismiss the message? With your one person GEDCOM, it goes through to completion and displays the following message:

After I captured the message I totally forgot to keep running it. The good news is it finished with the Gedcom with only Raymond. However, when I ran it on the full gedcom I got the following:
See next reply for 4th attachment

4th attachment

I seem to be quite forgetful today. I also got messages that it didn't like the NewspaperPlace and PubPlace for
Template Record: 6
Source Record 328

OK, good progress, thanks. If RM is going to throw these wobbles in its GEDCOM, it might be better to save a list of suspect fields as a table, and present it in the Records Window when the plugin has completed. It's not a difficult change. I'll see if I can add more details, such as the fact that the citation is against, to make it easier to manually check the GEDCOM file.

The Meta-field failure means that the plugin could not create a shortcut for adding ItemID data to a source based on Template #72. This aborts updating the Citation data and displays the second critical error message warning the user to undo the updates. You didn't say that the plugin crashed after this, so I will assume it stopped cleanly.

If you took the advice to undo changes, could you run it again please, and review the structure of Source Template #72 (this is the Record Id, not the RM template number). Are the data fields and their definitions the same in RM and FH? If there is a discrepancy, the first prototype would not have spotted it as it doesn't do anything with the templates. It's only once we start using them that any mismatch in structure comes to light.

Ditto for Source Template #6 if that was also the same Meta-field error, please.

I have an appointment in a bit. When I get back would it help if I make a bigger gedcom and strip out the living? And then email it to you in case I miss something.

For both #6 and #72 the fields are defined the same way.

#72
In FH:
<{Subject}, ><{Title}, ><{CreationDate}, |undated clipping, ><from unidentified newspaper;>< {Collection}><, {Provenance}>; <privately held {YearOwned:Year}> by <{Relationship}, ><{LastKnownOwner}, ><!{StreetAddress}|{<sc>address for private use</sc>}><, {CityAddress}><. {Annotation}>.

In RM:
<[Subject], ><[Title], ><[CreationDate], |undated clipping, ><from unidentified newspaper;>< [Collection]><, [Provenance]>; <privately held [YearOwned:Year]> by <[Relationship], ><[LastKnownOwner], ><![StreetAddress]|[<sc>address for private use</sc>]><, [CityAddress]><. [Annotation]>.
#6
In FH:
<{Author}, ><"{Article}," ><{Type}, ><i><{NewspaperPlace}|{Newspaper}></i><, {IssueDate}><, {Pages}>; database and images, <i>Ancestry.com</i> (http:///www.ancestry.com< : viewed {AccessDate}>)<, {Collection}><; {CreditLine}><. {Annotation}>.
In RM:
<[Author], ><"[Article]," ><[Type], ><i><[NewspaperPlace]|[Newspaper]></i><, [IssueDate]><, [Pages]>; database and images, <i>Ancestry.com</i> (http:///www.ancestry.com< : viewed [AccessDate]>)<, [Collection]><; [CreditLine]><. [Annotation]>.