* Import of RM7 Citations

[fhtess65]

@Teresa, the finished product (hopefully by end of this week) will import all your custom RM templates directly, along with re-creating all the source data granularity that is lost on the normal GEDCOM import. Going forward, you can create new templates directly in FH. It does everything that RM does, and some more!

Thanks, Mark - that's great...I'll definitely have a boo when it's ready.

And yes, I have also created templates in FH. I love each program for various reasons, so maintain trees in each. Lots of work, but I also am all about not having all my eggs in one basket.

If I were just starting out, doubtless I would choose FH as my primary program.

Teresa

[fhtess65]

So that offers little hope of assistance with FM imports of RM Templates, which if FH offered Direct Import from RM then CP would have to consider how to handle RM Citation metafields in the RM Bibliography. The only chink is they say they will investigate and check it out.

I'm a "bad" genealogist, and rarely use that Bibliography field as I don't cite by EE, more a hybrid of Strathclyde and EE, favouring Strathclyde. I'll be curious to see how this all works.

[tatewise]

Teresa, for most scenarios it will work well, especially if you make little use of the Bibliography.
However, the publically published plugins needed to work for as many scenarios as there are users.
So any limitations need to be made clear to users of the Bibliography that include Citation-specific Metafields.
Source record Metafields are perfectly OK.

[fhtess65]

Teresa, for most scenarios it will work well, especially if you make little use of the Bibliography.
However, the publically published plugins needed to work for as many scenarios as there are users.
So any limitations need to be made clear to users of the Bibliography that include Citation-specific Metafields.
Source record Metafields are perfectly OK.

Yes, I understand that. It's always up to the user to decide if the plug-in offers the usability they require and to take appropriate cautions when using it. If I didn't work full-time, I'd be building way more templates in FH than I already do. Just not enough days in the week or hours in the day...plus, my husband does like to see me from time to time when I'm NOT doing genealogy

[Mark1834]

The plugin is now ready for more general user testing. I have developed it using a tiny database that I constructed in RM, plus the FH Sample Project exported to RM, then imported back again. Everything seems to work as intended, but how well does it cope with real RM projects?

To recap, all sources in RM are based on templates, either one of the 400+ supplied with the program, or those customised by the user. Unfortunately, when FH imports a GEDCOM file originating in RM, everything is converted to what FH7 calls generic sources. The custom template definitions are discarded completely, and while all the source data are imported, the structured version is hidden in Undefined Fields, and only mashups are visible.

This plugin does three things:

1. By reading the GEDCOM file that you used to transfer your database to FH, it extracts all the user-defined template definitions, and re-creates them as FH Source Templates. These templates should be virtually identical to the RM originals, as the basic structure of templated sources is very similar in the two apps. Footnote and bibliography definitions are copied across verbatim, with only minor translation (replacing [] with {}).

2. RM does not export its built-in template definitions, so we have to reconstruct them from the data included in the database. Every field used is included, along with an approximate label. No type definitions are included, so everything defaults to plain text (other RM types are date, place and name), and there are no data prompts. These should be reasonably clear, but you can always tidy them up in FH afterwards if you wish.

3. Finally, all sources are linked to their original templates, and the source data moved from Undefined Fields back into its correct place in the individual source fields. All other project data are left unchanged. This means that it doesn't matter if you have updated your project since your original import. Nothing entered by you is overwritten or deleted. The UDF fields are then deleted, but only after successful copying. Source level fields (yellow background in RM) are added back into their original sources, and citation level fields (green background in RM) are added to the appropriate source citation for the referenced fact. I have chosen not to update source titles, but that can be changed if necessary.

This plugin does not

A. Translate Bibliographies and Footnotes completely from RM to FH format. Hopefully the basic changes made create at least useable versions, but how good are they in the real world? I am not intending to create a "universal translator" (we'll leave that to Dr Who), but if there are any simple changes we can make that would be of use to the general user, I'm happy to add them.

B. Restructure the splitting/lumping of your data. This is intended as a general plugin, not tailored to specific users. If you use a bibliography layout that is not permitted by FH (e.g. including citation fields), you will have choose between modifying your data structure or changing the bibliography format. That is your choice, and the plugin doesn't make it for you.

Feedback welcome - what I'm aiming for is something that does most of the formatting for most of the users, leaving you to fine tune details manually. Realistically, it's a plugin that will be run once only, and only by a limited number of people.

As ever, it is best to try it out on a copy of your project, not the original. You can always select Edit > Undo Plugin Updates from the FH menu if you don't like the results, but better safe than sorry!

(Attachment deleted - replaced by version 2, below)

[fhtess65]

Thanks, Mark - I'll have a go this weekend and see what happens. Sounds very interesting and I'm curious to see the results

Teresa

The plugin is now ready for more general user testing. I have developed it using a tiny database that I constructed in RM, plus the FH Sample Project exported to RM, then imported back again. Everything seems to work as intended, but how well does it cope with real RM projects?

<SNIP>
Feedback welcome - what I'm aiming for is something that does most of the formatting for most of the users, leaving you to fine tune details manually. Realistically, it's a plugin that will be run once only, and only by a limited number of people.

As ever, it is best to try it out on a copy of your project, not the original. You can always select Edit > Undo Plugin Updates from the FH menu if you don't like the results, but better safe than sorry!

[BakerJL75]

Real world data problems. I exported my RM7 database and imported it into FH7. Then I ran the plugin and got:

Screenshot 2021-04-29 085708.png (42.64 KiB) Viewed 4710 times

I chose not to rollback the changes, but as far as I can tell, the plugin did nothing.
I'll see if I have any clue about a nil HINT field.

[BakerJL75]

This may help. I use both brief and long hints a lot.

Screenshot 2021-04-29 090312.png (215.24 KiB) Viewed 4701 times

[Mark1834]

Sorry about that. I'll check it out later when I get back home. The message is telling me where to look.

[BakerJL75]

Sorry about that. I'll check it out later when I get back home. The message is telling me where to look.

No apology needed. Bugs are part of coding. I'm just grateful you're doing this.

[tatewise]

Mark, if this helps it is to do with appending HINT and LONGHINT when there is no HINT.
The fix is I think:

Code: Select all

		if Field.HINT then
			p = fhCreateItem('PROM', pF)
			fhSetValueAsText(p, Field.HINT)
		end

		if Field.LONGHINT then
			if fhGetTag(p) == 'PROM' then
				fhSetValueAsText(p, Field.HINT .. ' ' .. Field.LONGHINT)
			else
				p = fhCreateItem('PROM', pF)
				fhSetValueAsText(p, Field.LONGHINT)
			end
		end

[Mark1834]

Sorted. If both HINT and LONGHINT are present, it concatenates them for the FH hint in the form "hint, longhint", which I think looks neater in typical examples than just a space. If only one is present, it uses that one as the FH hint, whichever way around it is.

Updated version attached.

[BakerJL75]

Progress! I look some more later, but it looks like it brought in all my custom templates. I'll play with them a bit later. However, it didn't do well on the RM Templates. It brought in #139 and #172, but they were incorrect. The fields were brought in, but the templates had data. I'll post the error, and then what one of the templates look like. After those two it gave the error message below.

Screenshot 2021-04-29 155644.png (41.86 KiB) Viewed 4509 times

Screenshot 2021-04-29 155705.png (61.64 KiB) Viewed 4509 times

[Mark1834]

Thanks Jackie. There are two separate issues here. Firstly, the Bibliography and Footnote definitions for built-in templates are not included in the GEDCOM export, so I have used an associated source to create an example. On reflection, it might be better to omit them altogether. I anticipate that the typical RM user would have used only a small number of the hundreds of examples provided, so while they are copying the original template title across, they might as well copy the Bibliography and Footnotes as well if they want to keep them.

The error you reported comes at the final stage, as the UDF fields are copied back to the sources. I suspect it may be related to blank fields or missing data, so I probably need to build a bit more checking into the code. I'll look at it this evening, and hopefully third time lucky!

[BakerJL75]

Thanks for looking. As far as me personally, I almost never use the standard templates, so I'm happy with it just bringing in the custom templates. But if you are going to publish it in the store, others may make much more use of the standard templates.

[Mark1834]

Latest update is attached.

I have not tried to recreate Bibliography and Footnote entries for built-in RM Templates. Any that are included in project Source Records are now left as Undefined Fields, so you can copy them into your new templates and modify as required if you want to keep them.

There is now more extensive checking for error conditions, which should help prevent unexpected crashes. It also detects if it is being run for a second time on the same project, and avoids creating duplicate templates.

There was also a mistake in one of the template selecting routines that meant templates were not always being selected correctly. RM template identifiers are all numbers (<10000 for built-in, >= 10000 for user-defined), but numeric data is generally stored in FH as text. This doesn't usually matter, but when checking equality, both sides must the same variable type. At one point in the script this was not the case, so templates were not being identified correctly, resulting in the error reported above. Hopefully, now fixed.

[Ruth]

Thanks Mark. I am a new user of FH7 coming over from RM. This will be a big help to me.

[Mark1834]

Welcome to FHUG Ruth. I assume from your profile that you looked at FH6 a while ago, but have now made the move. FH7 is a much better fit for RM users than FH6, and you might find the related thread on importing the RM To-Do List useful as well. Neither templated sources nor research notes are available in FH6.

[Ruth]

Thanks Mark. Yes I had previously reviewed vs 6, but found it a bit daunting. I decided to give vs 7 another look, and really liked what I saw. I am slowly learning the different features, and am really looking forward to using this software.
Ruth

[tatewise]

Ruth, could you please update your Profile to the right of your posting to say Family Historian: V7
Click your username, chose Edit profile, select FH V7, then scroll down to Submit button.

[BakerJL75]

Tested .3
As before it did fine with all my custom templates, but none of the built in ones came in.

Screenshot 2021-05-01 132649.png (62.53 KiB) Viewed 4309 times

[Mark1834]

Sorry about this Jackie. I can see what has happened here, but I’m not clear why. The script is reading in your sources, but it can’t find an identifier for the template. The code looks ok, so my assumption about how RM structures its GEDCOM must be incorrect. I don’t have a real database to test on, but I’ll experiment tomorrow.

Which built-in sources do you use please, so I can try to replicate your sources as closely as possible?

[BakerJL75]

Which built-in sources do you use please, so I can try to replicate your sources as closely as possible?

I don't think this will help much. I switched to entirely custom templates a long time ago, because I always want to change something later. The only built in source I've actually used is Newspaper, Online Archives. And only for source.

[Mark1834]

Thanks Jackie. The only way I can reproduce the error is by modifying the GEDCOM file before running the plugin to force an error condition, so can I ask for your help to diagnose what is happening please? I don't want to post multiple test copies of the plugin, but if you could add one additional line of code, it would tell us which source is causing the problem.

Starting from a fresh test database, could you open the plugin window, and instead of running the import plugin, click on More>> at the bottom of the window, and Edit... to open the plugin in the debugger.

Scroll down to the line where the problem occurs, 345, and add the following line of code between lines 344 and 345 (Lua doesn't worry about whitespace, so blank lines and indentations have no significance, they are just there to help readability).

Code: Select all

if not tonumber(TID) then fhMessageBox(fhGetDisplayText(pS) .. ' - ' .. fhGetDisplayText(pT) ) end 

It should then look something like this

Capture.PNG (11.87 KiB) Viewed 4229 times

Run the modified plugin by clicking on the Go button at the top of the window (you do not have to save it first).

The error is caused by the template identifier (TID) not being a pure number (either missing or containing non-numeric characters), so the tonumber() conversion returns a nil result. The new line of code will display the name of the errant source and the TID, as in this example where I created a new Newspaper Source and manually deleted the TID prior to running the plugin to force the error. When you click on ok, the script will resume and crash.

Capture1.PNG (2.75 KiB) Viewed 4229 times

What message do you get please? Once we know what is causing the problem, we can move on to why and fix it.

[tatewise]

Just a couple of comments.

You could check if pT:IsNull() that would indicate there is no TID tag at all.

I noticed that there is a lot of management of table values to check for uniqueness or existence in another table.
Have you considered using the value as the table key instead of the table value?
i.e. Instead of table.insert(tblA,Value) use tblA[Value] = true
That ensures there is only one entry for each unique Value and avoids the need for function UniqueValues(tblA) and probably function InTable(val, tblA).
That is the conventional Lua technique for building a dictionary of unique table entries to avoid searching tables for values.