* Import of RM7 Citations

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
Post Reply
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Import of RM7 Citations

Post by cwhermann »

I am new at writing plug-ins and the first attempt is the development of a plug-in to import citation citations generated and exported in RM7 to an "equivalent" citation in FH7. The goal is to have one plug-in for both Type 1 (or splitter) and Type 2 (Lumper) citations and both FH7 built in citations and custom templates.
Would like some input from folks with a lot more Gedcom file, database and plug-in experience and knowledge:
with the plug-in, I am attempting to search the FH7 Source Records and the UDF file for source/citation fields, exported by RM7, extract the text from the field and import it to an equivalent field in a FH7 citation.
For custom templates, users will need to create a custom template in FH7 that mirrors the RM7 template as in the attached images.
Attachments
RM7 Template.PNG
RM7 Template.PNG (29.72 KiB) Viewed 7766 times
FH7 Template.PNG
FH7 Template.PNG (38.41 KiB) Viewed 7766 times
Curtis Hermann
FH 7.0.15
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

My advice would be to take it one step at a time. Make sure you understand what simple scripts are doing first.

The basic introduction in plugin help is quite good, so use that to start building simple scripts for the Sample Project. Produce a list of all individuals, using the looping techniques described in the help. List all sources. List details within the source. As you understand one step, build in a bit more complexity. Even relatively simple sounding tasks like that will give you the basics of how the data are structured, data references, and links between records.

Once you understand how to list records and the data within them, you can move on to creating and deleting new records and links. But take the time to fully understand the basic listing first. You will need that grounding for the next step.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

I go along with Mark's advice. There are two main aspects that need to be grasped:

(1) Lua language concepts and library functions.
In the Plugin editor use Help > Lua Online Reference Manual (v5.3.5) for a formal specification.
That defines the programming language features that underpin Plugin scripts.

(2) FH functions and methods
In the Plugin editor use Help > Plugin Help and Help > Family Historian API.
They define the interface to the Project database and rely heavily on Data References.


There are further resources to draw upon but we can investigate those later.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Before I wrote the post, I had been through the FH Plugin Help- How to Trite Plugins and the exercises & Intro to Lua, The Lua 5.3 Ref Manuel and the KB. I think these resources assume the reader has some basic level of understanding of coding and the FH API, that I don't have. Things like it appears there is a hierarchy in the code based on indentation/tabs. Is this required or is it just convention to help the writer keep track of "steps" in the code? Some lines of code contained different types of brackets. What is the functional difference between curly brackets { }, regular brackets ( ) and square brackets [ ]? What is the difference between two periods . . and three periods . . . and their location. The same for single = sign and double == equal sign.

Anyway, I thought if I could clarify what exactly I want the plugin to accomplish, I might be able to correlate that to some of the code I was looking at. Hopefully I can find a "Lua for beginners" on Amazon so I can understand more of the Reference Manual and Pugin Help.
Curtis Hermann
FH 7.0.15
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Import of RM7 Citations

Post by ColeValleyGirl »

cwhermann wrote: 10 Apr 2021 14:24 Hopefully I can find a "Lua for beginners" on Amazon so I can understand more of the Reference Manual and Pugin Help.
Try Programming in Lua Fourth Edition in the Amazon site local to you.

You will probably also want/want instead a beginners guide that doesn't assume a knowledge of programming. They exists but I don't know which one I'd recommend (never having used one -- I cut my programming teeth in the late 70s). You should pick one that isn't tied to a particular environment (like Minecraft).
User avatar
mjashby
Megastar
Posts: 719
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Import of RM7 Citations

Post by mjashby »

This online tutorial seems reasonable, but haven't had time to follow it through very far yet.

https://www.tutorialspoint.com/lua/index.htm

Mervyn
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Unfortunately, there are not that many good Lua beginner's guides around in my experience. It is not a very common first language, so outside FH, most people probably come to it with some coding experience already, so the documentation can get very technical very quickly. There was talk here a few months ago about building some "coding for beginners" material into the KB, but nothing has come of it yet.

It's very common these days for "Introduction to Coding" books and websites to use Python rather than Lua. The basic concepts are very similar, and even a lot of the detailed syntax is common. It might be worth spending a bit of time learning the basics of variables, operators, functions, etc, using Python. There is a vast amount of suitable material material available online and in book form for Python, and the language itself is a free download.

Once you've got the basics of how a typical language works, you can apply that to the detailed syntax of Lua.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Curtis, to answer some of your specific questions:
Things like it appears there is a hierarchy in the code based on indentation/tabs. Is this required or is it just convention to help the writer keep track of "steps" in the code?
It is not required. The code can be completely free format. It is a convention that helps the author.
Some lines of code contained different types of brackets. What is the functional difference between curly brackets { }, regular brackets ( ) and square brackets [ ]?
curly brackets { } define tables or lists such as { red; green; blue; yellow; } although they can get more complex than that.
regular brackets ( ) usually enclose parameters or arguments supplied to a function, e.g. Compute( one, two, three ).
square brackets [ ] usually hold an index to a list, e.g. [3] would select blue from the list above, or in FH data references will select the 3rd instance of a tag such as the 3rd OCCU (Occupation) fact.
What is the difference between two periods .. and three periods ... and their location.
two periods .. is one of the many operators including + (add), - (subtract), * (multiply), / (divide) e.g. 2 + 3 - 9
.. is the concatenation operator that joins two text strings together e.g. "abc" .. "xyz" becomes "abcxyz"
three periods ... apply to regular brackets ( ... ) and say that any number of parameters or arguments may occur.
The same for single = sign and double == equal sign.
single = sign is the assignment operator, e.g. answer = alpha + 1 means answer is assigned the value of alpha plus 1.
double == equal sign is the equals comparison akin to < and > comparisons, e.g. if x == y then
Anyway, I thought if I could clarify what exactly I want the plugin to accomplish, I might be able to correlate that to some of the code I was looking at.
I see the Plugin operating in two major phases.
(1) Scan all the records and fields to gather RM _TMPLT FIELD NAME details and where they are used and save in tables.
Somewhat like the Sample Plugin Scripts for Find Date Phrases and List All UDF Fields.
(2) Traverse the tables to create Source Templates and then update Source records and Citations.
The Sample Plugin Scripts for Create Family shows how to create records and fields.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

started through the link from Mervyn - very helpful - seems written for someone with little or no knowledge of Lua compared to their Reference Manuals.
Also ordered a couple of books from Amazon and received notification Getting the Most from FH6 has shipped.
Thanks to all
Curtis Hermann
FH 7.0.15
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Trying to lay out a game plan on how to attack this having reviewed the Lua programing materials.
Given that many of my citations are Type 2 or lumped - this will involve bringing in the Source and then each of the citations for an event tied to that source. Looking at the RM Gedcom, each source is identified with unique number in as @SXXXX@ with source level information including a title, Template ID, Field names-values, etc.
Each citation on the other hand is listed (tied) to the event, beginning with the source number, @SXXXX@, and then followed with the citation level details, filed names-Values, notes, etc.

Appears to me I will need two (or maybe a whole bunch more) passes through the RM Gedcom file, once to deal with the sources and then second time to deal with the citations.

Looking for guidance on best approach for first pass through the Ged file:
1) A loop to go through the RM Ged file to identify each Source record then then take each "attribute" (line in the Ged file) to create a new template and source record in FH - then go to the next source record and repeat
or
2) A loop to go through the RM Ged file to identify each Source record using it to build a table, with the @SXXXX@ as the key and each "attribute" as a value. And once the table is built then use it to build the FH templates and source records.
Attachments
RM GED Image.PNG
RM GED Image.PNG (40.97 KiB) Viewed 7577 times
Curtis Hermann
FH 7.0.15
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Curtis, you should avoid reading the GEDCOM file except as a last resort.
It is much easier to use the FH API functions to gather the information which can be done in a single pass.

Look at the Sample Plugin Scripts for Find Date Phrases and List All UDF Fields.
They both use the style of loop below to search the Project records for certain fields.
In our case, it needs to look for "SOUR" tags for Citations and Source records, and also "_TMPLT" tags for RM details.
One pass can collect the Citation details associated with Source Record Id, then later integrate that with details from the Source records with matching Record Id, and collate that against a particular RM Template Id ("TID").
It does involve the use of structured tables to hold the details and associate them together.

Code: Select all

tblTypes = {"INDI","FAM"} -- Scan both Family and Individual Record Types -- Our plugin would include "SOUR" records
ptrItem = fhNewItemPtr()
for iType, strType in pairs( tblTypes ) do
    ptrItem:MoveToFirstRecord( strType )
    while ptrItem:IsNotNull() do
        strType = fhGetTag(ptrItem)
        if strType == 'DATE' then -- Here our plugin would be testing for "SOUR" and "_TMPLT" tags

        end
        ptrItem:MoveNextSpecial()
    end
end
Try running the above script and see if you can extend it to detect "SOUR" and "_TMPLT" tags.
I can provide more detailed explanations and snippets of code if necessary.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Da - A light bulb moment !!!
The whole goal here is not to "re-import" portions of the GEDCOM file differently, its to "re-arrange" the information already imported by FH7. Hence the reason for trying to understand what got imported and where.
Thanks
Curtis Hermann
FH 7.0.15
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Import of RM7 Citations

Post by ColeValleyGirl »

Curtis, you should be aware that there's a library of useful functions for FH7 which might avoid you have to reinvent some wheels.

The documentation is here: fhUtils.

You make it available in a plugin by including this line near the top:

Code: Select all

require ("fhutils")
If you were writing a plugin that needed to be compatible with V6, you couldn't use it, but as Source Templates are new in V7, that isn't a problem.

In particular the functions allitems and records do the looping that Mike is talking about.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Yes, fhUtils is a good alternative but there are no examples of it in the Sample Plugin Scripts so I chose to start there.

Curtis, I'm glad you had that :idea: moment as it avoids me needing to explain why you cannot work on the GEDCOM file.

A useful technique for designing software is HIPO = Hierarchical ~ Input ~ Process ~ Output where hierarchically nested tasks have their input details, processing rules, and output details defined.
Until yesterday, I had thought the only input details were the RM _TMPLT.FIELD data imported to the FH Project and that one key task would analyse those and reverse engineer the FH Source Templates as output details.
However, we now know that there are RM Template definitions in the GEDCOM file to help create FH Source Templates.
So the top-level HIPO tasks now look like this:
1) GEDCOM RM Source Template _STMPLT ~ Synthesise FH Source Template ~ FH Source Template & TID cross-ref
2) RM Source RM _TMPLT.FIELD & TID cross-ref ~ Synthesise Source Metafields ~ Updated FH Source Records & RID cross-ref
3) RM Citation RM _TMPLT.FIELD & RID cross-ref ~ Synthesise Citation Metafields ~ Updated FH Citations
Those cross-ref details tabulate the Source Record Id (RID) against the Source Template (TID) and the Metafields.

There are aspects that are unclear to me involving the Citation Where Within Source PAGE, and Source record TITL, ABBR, _SUBQ & _BIBL fields and how the FH Source Template Title, Footnote, Short Footnote & Bibliography Formats are derived.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Import of RM7 Citations

Post by ColeValleyGirl »

tatewise wrote: 12 Apr 2021 09:35 Yes, fhUtils is a good alternative but there are no examples of it in the Sample Plugin Scripts so I chose to start there.
The two examples I suggested have examples in Snippets in the KB

Loop All Items (allitems)
Loop Through Records (records)
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Just to be clear, I think those Snippets illustrate the pre-fhUtils vanilla Lua functions as they say Requires: None.
Once the user has used require ("fhutils") then only script based on the Usage examples is needed, but with the library prefix, i.e. fhutils.allitems( ) or fhutils.record( )

Maybe those Snippets need updating with FH V7 fhUtils examples, eventually :D
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Import of RM7 Citations

Post by ColeValleyGirl »

Eventually, those snippets and others will be annotated to say that they're superseded for V7 by the fhUtils or fhFileUtils library (with pointers to the documentation, which -- in the case of allitems and records -- has an example of use).
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Mike Wrote:
There are aspects that are unclear to me involving the Citation Where Within Source PAGE, and Source record TITL, ABBR, _SUBQ & _BIBL fields and how the FH Source Template Title, Footnote, Short Footnote & Bibliography Formats are derived.
Here is what I found with my custom template citations created in RM.
The RM Source Record TITL field is a text string of the footnote created in RM less the source and citation template field codes. This is imported to the FH Source Record Title field, the Footnote, Short Footnote, and the Bibliography.
The RM Source Record ABBR field is what RM calls the Master Source Title and is imported to the FH Short Title Field. If I would creating the Source Record in FH, I would use this as the Title not Short title.
The RM Source Record _SUBQ field is the text string of the short footnote created in RM less the source and citation template field codes and is imported to the UDF file.
The RM Source Record _BIBL filed is the text string of the bibliography created in RM less the source and citation template file codes and is also in the UDF file.
The RM Citation PAGE field is a text string of all the citation level template field values. This is imported to the FH Where Within field, the FH footnote and short footnote.

I have not traced a RM Free-form citation "through the process" as yet.
Curtis Hermann
FH 7.0.15
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Curtis, I was coming to a similar conclusion with the consequence that those fields pose us something of a problem.

One thing to point out is that the Source record TITL cannot be set to the ABBR value.
In FH (just like in RM I suspect) the TITL field is automatically derived from the Title Format codes in the Source Template.
So whatever text is assigned to the TITL will soon get overwritten by the Source Template formatting process.

It would be useful to know the RM source and citation template field codes so we could automatically create a matching set of codes in the FH Source Template Format fields.
However, those RM source and citation template field codes are omitted from the TITL, _SUBQ & BIBL fields.
So the plugin will have to analyse the text in those fields and match up each fragment with FIELD Metafield text values.
Then the Names of those Metafields can be synthesised into the Source Template's four Format codes.
That sounds like fun :roll:
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Now that I have mastered Lua ;) , I'll dig into the API, HIPO, and fhUTILS - and we are not even half way through the alphabet yet!!!

For someone completely new to LUA, I found the link Mervyn provided, https://www.tutorialspoint.com/lua/index.htm very helpful and recommend anyone just starting with plugins may want to go through the tutorial before the FH Plugin Help and KB.
Curtis Hermann
FH 7.0.15
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Import of RM7 Citations

Post by ColeValleyGirl »

Curtis, that's useful to know -- I've updated Getting Started Writing Plugins to reference it.
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Mike wrote:
It would be useful to know the RM source and citation template field codes so we could automatically create a matching set of codes in the FH Source Template Format fields.
However, those RM source and citation template field codes are omitted from the TITL, _SUBQ & BIBL fields.
So the plugin will have to analyze the text in those fields and match up each fragment with FIELD Metafield text values.
Then the Names of those Metafields can be synthesized into the Source Template's four Format codes.
I think we have the RM source and citation template field codes. Unlike the FH Source Template edit window, the RM template window does not have a column titled "code", the field code is the field name in [ ]. See image in my April 9th post. The field names are in the FH Source and Citation Records so we have the info in FH to create a portion of a source template.
If I understand Mike's to-level HIPO steps in his 12 April post, item 1) is to get the template info from the RM GEDCOM_STMPLT which in addition to the field names, also has the Footnote, Short Footnote and Bibliography templates. Since the RM FIELD names == FIELD codes, and we have the FIELD Metafield text values, (albeit in UDF file) there would be no need to analyze the TITL, _SUBQ & BIBL fields and match up the fragments with the FIELD Metafield text values.

I am still trying to learn the API and how to manipulate the data already in FH and was attempting to create an initial Plugin just to get the Source Record info into a FH custom template. I manually created a custom template to match/duplicate a RM template, but came across a couple of issues. How do I match the Source Record to the correct template and a Bibliography format issue.
The RM GEDCOM_STMPLT record includes TID Field which is imported to the FH Source record and for my simple exercise to link this to the template, I included that in the template description, hoping I can figure out how to search for it in my Plugin.
Secondly, in creating the custom template in FH, it appears any field/field code designated as citation level entry cannot be used in the Bibliography sentence template. When I "save", I get an error message "Bibliography Errors: Citation field code not valid in this context: {field code}. It allowed me to save anyway, but when I created a new citation using the template, the citation level fields were not shown in the Bibliography. I have looked through the Help and KB, but see this mentioned specifically. Is my interpretation correct? If so, this is something that will complicate the creation of the FH templates from the RM GEDCOM_STMLT.
Curtis Hermann
FH 7.0.15
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Curtis says:
the RM GEDCOM _STMPLT in addition to the field names, also has the Footnote, Short Footnote and Bibliography templates
That would be excellent, but the GEDCOM posted by Mark here Re: Freeform citation in V7 ~ Sun 11th Apr 2021 20:58 does not contain any Footnote, Short Footnote and Bibliography templates.
Please can you post a fragment of RM GEDCOM that does contain an example of those templates.

Yes, the TID will be the cross-reference between the Source Template and the Source records that use it.
For the moment, with your manually created Source Template, the plugin must loop through Source Template (_SRCT) records using the Loop Through Records method until it finds the one with the Name that includes the TID.
Then the Source record needs a _SRCT tag to be created and the link pointer to the Source Template set as its value.
That is how the Source Template metafield definitions and footnote formats are associated with the Source record.

In the real plugin, HIPO 1) creates the Source Template, so it knows the pointer which can be saved in a {list} table indexed by the TID and forms part of the Output Details, e.g. tblSRCT[TID] = ptrSRCT cross-ref
HIPO 2) updates the Source records and gets the TID entry from {list} table to set _SRCT tag, e.g. ptrSRCT = tblSRCT[TID]

You are correct in that Citation-specific Fields are not allowed in the FH Bibliography Format. I don't know why.
If allowed, it would lead to an entry in the Bibliography list for every separate Citation, which maybe CP don't like.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

tatewise wrote: 14 Apr 2021 16:05 but the GEDCOM posted by Mark here Re: Freeform citation in V7 ~ Sun 11th Apr 2021 20:58 does not contain any Footnote, Short Footnote and Bibliography templates.
Those fields are in the Source Template definition window (off to the right of the area I captured), and are plain text fields. Unlike FH, there is no assistant or wizard to guide data entry, so we'll have to leave it to someone who speaks RM to define those.
Mark Draper
User avatar
cwhermann
Famous
Posts: 155
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann »

Here is image of 0 _STMPLT for the template used in for the Source/Citations in my previous images so there is consistency across the images. In RM there are no default Footnote, Short Footnote, and Bibliography templates - they are blank unless the users builds/creates them. Mark's GEDCOM file would not contain them unless he took the time to create them when he did his short and quick experiment/trial.
I think everything is there to build a template in FH. In addition Footnote, Short Footnote, and Bibliography, all the FIELD info (name, display, hints, type, etc) is there. It's just that at my skill level, it's copy and paste, and I would still have to deal with switching the [ ] brackets to { } brackets. :D
Attachments
RM GED TID 10028 STMPLT.PNG
RM GED TID 10028 STMPLT.PNG (51.8 KiB) Viewed 7411 times
Curtis Hermann
FH 7.0.15
Post Reply