* Sample plugin for converting sources to use templates

[stanm]

I'm considering converting my database sources to use Advanced templates. I thought I would try to convert the easily recognizable source types, one at a time. The others I may just change manually.

I have a rough idea about how to do this, but it would be helpful and time saving if there was some skeleton code to start with.

Thanks!

[Mark1834]

I’ve been playing with an experimental script to format GRO Birth Index citations in a way that could be converted to a custom template. It doesn’t do the final conversion step (I’m not 100% convinced yet that’s the approach I want to take), but I’ll happily post up a draft in a day or two when it’s properly debugged.

Probably not exactly what you were after, but it might give you some ideas.

[shoshk]

Hi Stan,

Are you a lumper or a splitter?

Shosh

[stanm]

Thanks Mark, for offering to share your code.
It sounds like parts of it may overlap with my needs.

I've been looking at existing plugins to get ideas and learn a little Lua. This kind of plugin is very specific for each of us, so there won't be any generic solution.

[stanm]

Are you a lumper or a splitter?

Hi Shosh,
I am a splitter, but I know of a few sources in my GEDCOM that fall in the lumper category.

[shoshk]

Hi Stan,

If you are sure that you want to move to template-based sources, then, as a splitter, I definitely think that a plugin is the way to go. You are likely to have multiple sources which need the same treatment. And I agree, a generic solution is not likely to work for most.

Although I'm not working on a plugin at the moment, I'm interested in seeing what other people are doing.

I haven't yet decided if I want to take the plunge yet. I'm still trying to understand what the benefits are vs the cost of reduced portability with other genealogy programs.

Shosh

[Mark1834]

My draft script is attached. As written, it displays all births supported by either a GRO Birth Index record or a full Certificate. The index is a lumped source, with the where details in Where Within: and the what in Text from Source. For a certificate, I enter the where in Publication Info: and a transcript in Text from Source.

For me, it is only worth using templated sources in a large existing database if existing records can be retrofitted into the template, otherwise you end up with untidy and inefficient splitting. The GRO Birth Index is a good candidate, as it is well structured and use of a suitable custom template could provide the option to split out the various data items (district, date, recorded name, mother's maiden name, etc) into separate fields. This would make sorting or querying on specific items easier (although as a general observation, querying templated sources is significantly more complex than for generic ones).

This plugin can be used to split the details into individual fields, ready for importing into a templated source. However, I have decided against going down this route. This plugin (which was written in FH7, but works exactly the same in FH6) gives me all the extra functionality that a templated source would give, without compromising compatibility with other apps. If I were to use a templated source, I would want another plugin to convert the data back to a fully structured generic form for export (not just copied to notes), and that is a lot of work for little perceived gain.

Perhaps it illustrates a general Catch-22. The more structured your existing sources are, the easier they will be to convert en masse to a templated form, but if they are already well structured, do you need to compromise compatibility by going down the templated route?

An output screengrab is attached. Hopefully, the plugin will be easy to follow and crib from as necessary. As discussed in another recent thread, I much prefer slightly more verbose but easier to read scripts, so it is quite likely many of the steps here could be done in fewer lines and run a few ms quicker. However, please do point out any errors!

Couple of additional points that illustrate how specific these routines would have to be -

• My GRO citations are always in the form District, Date, Detail, with the actual content recorded in the style Mary Jones (Smith) for birth and Joe Bloggs, 56 for death. The string matching and splitting has to be customised to your style, and they all need to be the same.

• I record index date as yyyy.q for historic quarterly volumes, and yyyy.mm for more recent monthly ones (e.g. 1873.4 and 1999.01, respectively). I leave them as text here, as they still sort in date order, but further processing would be needed to convert them to actual dates, and this would need to reflect your recording style.

Capture.PNG (36.61 KiB) Viewed 5370 times

_GRO Birth Index.fh_lua

(4.56 KiB) Downloaded 145 times

[stanm]

I'm still trying to understand what the benefits are vs the cost of reduced portability with other genealogy programs.

Yes, portability is certainly a concern when it comes to the handling of sources. I use 3 other programs (in addition to FH) that each have a different scheme for source templates. Currently, I don't even attempt to move data between these programs. On the plus side, FH will generate various reports with footnoted sources cited in a desired format. And, unlike the other programs, FH is extensible through its Lua interface.

But, I'm still trying to size up the task before I take the plunge.

[stanm]

My draft script is attached. ...

Hi, Mark
Thanks very much for sharing your work.
I'll need to spend some time with it (step through it with the debugger & such) before I comment.

[shoshk]

Hi Mark,

I've taken a look at your code. It doesn't work for me, since my sources are set up quite differently. It runs, but returns an empty result set. So, I looked at it from a high-level code review perspective.

I have only a few comments. (I hope you will take them in the spirit in which they are intended)

1) Variable names like ptrI, ptrP, etc are not very descriptive. ptrINDI is much clearer, particularly for somebody else reading your code (and also for you when you want to make changes to your plugin several months from now).

2) In the same vein, I would use the values 'Index' and 'Certificate' for Type. It requires a little more typing, but it's much clearer.

3) The code starting from line 120-159 could be nested inside your if which starts at line 110.

In general, I like your coding style. I find it very readable.

Regards,
Shosh

[Mark1834]

Thanks Shosh, good comments. My house style is to use single upper case characters for flow control flags or variables of limited scope, but I agree longer more descriptive names would be clearer.

To be honest, I’d be surprised if it gave a full output without any modification. The routines for splitting up the citation and text into their component parts are only called once on each iteration around the loop, so could have been left in the main body of the program. However, putting them in functions at the top of the listing makes them easier to customise to the user’s own recording style, as all the parsing is fully encapsulated in the short functions, so it is only those few lines that need changing.

[stanm]

My draft script is attached. As written, it displays all births supported by either a GRO Birth Index record or a full Certificate. ...
This plugin can be used to split the details into individual fields, ready for importing into a templated source. ...

Hi Mark,
I used your plugin with one instance of a GRO Birth Index and one of a GRO Certificate. I took data from two rows of your capture image, so the source & citation would follow your expected format. I was able to follow your code with ease and display the query output for the two records. Your code format and indentation makes it easy to read.

Because I created the two test records incrementally, I did see crashes in ParseCitation() when SOUR.PAGE was empty and when SOUR.PUBL was empty, suggesting that ptrP needs to be checked after the ptrP:MoveTo() calls.

One thing that I might have done differently is use string.match with a regular expression that captures the extracted tokens, instead of using the functions ParseCitation() and ParseEntry().

Although I won't be using your code as a starting point, it has been helpful for learning. As a preliminary step, I plan to use the "Generic Type" field (SOUR._TYPE) to first categorize all the sources in my database. Then I'll target an "easy" category for a first pass at a conversion plugin.

I've been collecting relevant plugins from the "Store" and sample code snippets from various postings on FHUG. This has nudged me to look for an external editor, hopefully, one that supports lookup of FH API functions through Winhelp, and Lua syntax support. Have you used another editor in your plugin development? Visual Studio Code looks like it might be promising.

[Mark1834]

Thanks Stan - agree that the code would need more error checking if there is a prospect of fields being empty (which I know they are not in my database). The FH editor and debugger is extremely basic, so an improved external one would be welcome, but I don't know whether that is possible or not.

[ColeValleyGirl]

I use Zerobrane for plugin editing but it doesn't support lookup of the FH API functions (shuld be possible to write a plugin for Zerobrane to do it, but I've never found the time).

[shoshk]

I use PSPad editor. There are a number of articles in the Knowledge Base about various aspects of using it.

Shosh

[stanm]

I use Zerobrane for plugin editing ...

I use PSPad editor.

Thanks for the suggestions. I've installed both, as well as vscode. I will take each of them out for a test drive.

[JoopvB]

Just read the posts about converting generic sources to templated ones.

If still needed, my conversion plugin may be used as a kind of prototype. It's specific for my setup of V6 generic sources but structured in a way to make clear what the project dependent parts are and which code is generic (should not need modification).

The plugin assumes that the generic source type is the basis for the template to use (can be changed of course, but seems reasonable to adopt it as a conversion strategy).

The project dependent version is directed at my (Dutch) implementation but shouldn't be to hard to understand (ask if something is not clear).

In V6 I used the note field to store URL(s). The first one for the website of the text (transcription) version of the Text to Source and the second one for the scan of the source. The plugin checks to see if the URL's are in the note field and, if so, moves them to the URL fields I have defined in my templates. Again... can be changed of course.

If there is a text to Source available then that is used as much as possible to find the field values for the templated source. If that fails it falls back on the generic title field. Note that the function getValue allows a number of fieldnames as kind of synonyms (in priority order) to get the value from.

I'm planning to upgrade the function to be more versatile and less aimed at my situation, but time and priorities...