* Repository Not Tracked on Imported RM Source Template Definitions

[Roberto]

My database was successfully imported from RootsMagic into FH7, then 7.0.8.2.

Since then I have been cleaning-up / standardising my Sources and citations including the content of Repository fields in individual Sources. Because the Source templates that had been used were those from RootsMagic, updating took the form of typing or pasting Repository names, therefore there is no link to any Repository listed or newly created in the Repositories in the Records Window.

It appears that the direct import brought in the RootsMagic Repository records and linked Source records to them but changes since, e.g. adding a Repository name to a Source defined by a Source Definition Template from the RootsMagic Import set, have not linked.

Is there a way to create links from the Sources with newly populated or amended Repository fields?

Cheers
Robert

[Mark1834]

Unfortunately, there is a fundamental difference between the methods used in FH and RM for recording Repositories.

FH follows the GEDCOM model, where the Repository is a distinct record type. All uses of a defined Repository link back to the same single record.

RM does not use Repository records, but simply records the Repository as free text in the Source Template. There is no central record of the same Repository, and each Source Template has its own individual description.

It may be a case of creating Repository Records in FH, then working through your imported Source Templates one by one to delete the Repository data and link to the new Repository Records. Once you have done that, all Repository changes will only need to be made once, in the Repository Record.

[tatewise]

The moral of this scenario is to check a few changes are organised as required before performing wholesale changes.

Having changed a lot of Source records to have just a Repository text field, the only shortcut to converting them to linked Repository records is to write a Plugin that uses the text field name to identify the Repository record and convert from a Text field to a Repository field in each Source record.

[Mark1834]

I've been playing a bit more...

It appears to be a little more complex than I thought originally, as RM can hold Repository details in Source Template fields and/or separate Repository records. However, these are not Repository records as FH and GEDCOM would define them, but entries in the general Address table identified as Repositories.

I did manage to crash RM8 completely by creating Repository details in the source citation and linking to a defined Repository record .

I think we need some feedback from Robert and other RM migrants about how this is normally handled in real RM databases.

[Roberto]

Mark,

RootsMagic does contain both Repository fields in Source definitions plus Repository records that are linked to the Source definition but not to the Repository field itself so you can have the Repository field showing "ABC" but the linked Repository showing "TNA, London etc".

Mike's suggestion re a specific "one time" plugin to replace the Repository text with a Repository link seems a good solution.

I am off to research writing a plugin unless one pops "out of the sky".

Cheers
Robert

[Mark1834]

It will be a fairly simple plugin, so given this variability in how RM handles repositories, it will be better written by an RM user tailored to their data. Relying on the plugin factory in the sky is not sustainable as it will dry up eventually, so the more users who can generate their own the better.

[Roberto]

Thanks Mark. Actually, I agree with you.

Cheers,
Robert

[Roberto]

I have jumped into the Plugin world and have my first Lua script to resolve my data as discussed above.

I am happy to debug the script when I get it working BUT I just can't get past a Syntax error - "36: syntax error near 'strSourRepoName".

I am sure that it is staring me in the face but I don't see it despite reading the Fhug pages, the code from other Plugins, other sites and a language book. Having been a COBOL programmer in the late 1960s and 1970s I may be hamstrung with unseen and incorrect assumptions!

If anyone could point me in the right direction that would be great.

Cheers,
Robert

[ColeValleyGirl]

You've put 'If' instead of if (in a couple of places -- lines 36 and 44)

You've also missed out a 'then' on line 44.

Line 25 should be 'fhNewItemPtr' not 'FhNewItemPtr'


If you're going to make a habit of Lua, you might want to think about using a better editor than the one that comes with FH, that shows up simple errors without having to run the code -- have a look as Visual Studio Code (my recommendation) or ZeroBrane. vsCode showed me all the errors I've pointed out above.

If you do decide to adopt one of those two editors, ask here and I'll give some suggestions for configuring them.

[tatewise]

Yes, the point is that Lua is extremely case sensitive. Upper and lower case letters are entirely distinct.

[Roberto]

Thanks Helen. Staring me in the face as I thought.

Syntax free and with your advice re the missing 'then', I have started debugging as there was no surprise when it didn't do what I wanted straight off.

Being relatively new to FH, this exploration into Lua is to fix imported data errors of my making. FH and existing plugins seem to cover an amazing wide scope so any likelihood of making a habit of Lua is some time off but the suggestion re an editor is welcome.

Cheers
Robert

[Roberto]

While debugging I have found that the Repository text data can exist in %SOUR._FIELD[1]% or %SOUR._FIELD[2]% or %SOUR._FIELD[n]% depending on the imported RootsMagic source template rather than in "%SOUR.REPO[1]>NAME%" for standard FM Source data.

To determine which %SOUR._SRCT>FDEF[1].NAME% would be needed to identify the "Repository" text, I examined the expanded template definitions in Sources in the Records Window. Here I found that there was a lot of variety, for example in the Baptism template the Field Definition: Repository is the 5th Field Definition and the Repository (t) is the 4th data item but in the Military template the Field Definition: Repository is, also in this example, the 5th Field Definition and the Repository (t) is the 2nd data item.

This could be managed in a long but straightforward If ... then ... else statement but is this the right path?

I also tried to export my test FH project, as a Gedcom to a new RM database and direct import back to a new FH project. The result was that all templates were converted to Free Form and all data moved to the Notes area. Probably not helpful in linking text Repository data to Repository records.

I have been pleased to carry out this research as it has greatly improved my understanding of FH (I am using 7.0.11), but will probably cease further work and return to "normal" family history research without managing Repository records, as I have done for some years.

I am happy for any thoughts and comments.

Cheers
Robert

[tatewise]

Yes, the _FIELD tag is not very helpful but there is a neater solution. You can refer to templated fields using their name.
See FH Help page Understanding Data References under DATA REFERENCES AND METAFIELDS near the bottom.
For any Repository metafield the shortcut is ~RP-REPOSITORY
e.g.
%SOUR.~RP-REPOSITORY>NAME%
That should work for all templated Source records.

In general, round-trip export/import of GEDCOM files via different products does not work well, especially for non-standard GEDCOM extensions such as Templates.

[Mark1834]

Source templates is an interesting one. CP clearly had sight of the detailed structure of the RM version when designing the FH equivalent. They could have chosen to use the same custom GEDCOM tags for much improved compatibility. Not doing so was probably influenced by both technical and commercial considerations.

[Roberto]

Thanks Mike,

However I must be still missing something. A simple query of:
%SOUR%
%SOUR._SRCT>NAME%
%SOUR.~RP-REPOSITORY>NAME% or %SOUR.~RP-REPOSITORY[1]>NAME%
returns Columns 1 and 2 populated but nothing in Column 3 so I have not gone on past this to the script to amend the entire file..

I apologise for the drawn out handling of this matter as until quite recently we have been supporting an adult son with terminal bile duct cancer through his final stages and that has absorbed much of my attention.

Cheers,
Robert

[tatewise]

I'm sorry to hear about your adult son.

I have attached a screenshot of the Columns tab of a Source Query below, and that works for me.

Presumably, the Sources listed in your Query do actually have a template Repository Field as opposed to a standard generic Repository field %SOUR.REPO>%

If you still cannot get any success then I suggest you attach a copy of your Query to your next posting.
Use File > Import/Export > Export > Query..., select query to export, click Export, and Save it somewhere memorable.
Then in the ATTACHMENTS tab of your FHUG posting use the ADD FILE option and choose your saved Query file.

QueryRepositoryField.png (43.14 KiB) Viewed 2145 times

[Roberto]

Thanks Mike. We aren't too old to know that the sun does come up in the morning.

Your query looks the same as mine, but mine is attached. I have also attached a sample of a source from the FH Project file to assist plus part of a screen shot from the Source Records window showing the template definition.

Cheers
Robert

[Roberto]

Mike,

I should have expanded the Repository field on my screen shot so here it is:

[tatewise]

The problem is that your Repository Field is defined with Type: Text i.e. a simple plain text field.
So its Data Reference shortcut is %SOUR.~TX-REPOSITORY%
Furthermore, it does not link to a Repository record.

Actually, all those template field definitions are Type: Text therefore...
1 _FIELD TX-NAME_OF_PERSON does not link to an Individual record
1 _FIELD TX-EVENT_DATE is not treated as a Date
1 _FIELD TX-REGISTRATION_DATE is not treated as a Date

I'm not sure whether you wish to 'improve' those templates, and if so how best to go about doing that.

[Roberto]

Thanks again Mike. The Data Reference shortcut %SOUR.~TX-REPOSITORY% works as expected in a Query.

My objective is to link Source records to the relevant Repository record. The method, based on your earlier suggestion to write and run a one-time Plugin, is to use the TX-Repository field to identify a Repository record and then convert that text field to a link to the Repository record and where no Repository record exists, to create one (and link to that).

I plan to then use only FH templates going forward so I have no thought to 'improving' the imported RootsMagic templates.

Based on my limited FH knowledge, I am not sure of the significance of your response that other fields as Text are either not linked to an individual record or are treated as dates. I appear not to have had any issue with reports that I have generated, with Sources lists linked back to Facts nor with Bibliography(s). Is the concern that text fields in the imported Source record may have dates that are different from the Fact date which the Source record supports? Am I getting the hang of FH now? If so, how big a problem will this be for me in the future, and, do I need to address it now before I plow on with new research?

Robert

[tatewise]

My objective is to link Source records to the relevant Repository record. The method, based on your earlier suggestion to write and run a one-time Plugin, is to use the TX-Repository field to identify a Repository record and then convert that text field to a link to the Repository record and where no Repository record exists, to create one (and link to that).

You will need to add a Repository field with Type: Repository and a shortcut such as %SOUR.~RP-REPOSITORY% so that the Plugin has somewhere to save the Repository record link and then delete the %SOUR.~TX-REPOSITORY% Text field.

Based on my limited FH knowledge, I am not sure of the significance of your response that other fields as Text are either not linked to an individual record or are treated as dates. I appear not to have had any issue with reports that I have generated, with Sources lists linked back to Facts nor with Bibliography(s). Is the concern that text fields in the imported Source record may have dates that are different from the Fact date which the Source record supports? Am I getting the hang of FH now? If so, how big a problem will this be for me in the future, and, do I need to address it now before I plow on with new research?

Text fields allow any text to be entered. So instead of the Name of Person you could enter 'Dallas, Texas, USA' or instead of an Event Date you could enter 'John Smith'.

A %SOUR.~RP-REPOSITORY% field of Type: Repository, as discussed above, provides a Repository record link.
Similarly, a %SOUR.~NM-NAME_OF_PERSON% field of Type: Name provides an Individual record link to the person.

A field such as %SOUR.~DT-EVENT_DATE% of Type: Date only allows valid dates in FH/GEDCOM format to be entered.
They can then be manipulated as Dates, i.e. sorted with other Dates, and displayed in various Date Formats.
e.g. 20 April 2022, 20/4/2022, 2022-4-20

[Roberto]

I am a lot happier with my understanding now.

My Options appear to be:
1) Write and run one or more Plugins to change imported RM Text data associated with all imported templates to be linked items, where appropriate in FH, plus change the imported template definitions to reflect the new dataset and avoid potential issues in the future.
2) Run one or more queries to show the content of the imported Text data to enable visual review and manual adjustment for consistency of content and presentation i.e. general tidying up of data.
3) Do nothing.

I now plan to take Option 2 and move forward.

Thanks everyone, especially Mike, for their invaluable support and guidance.

Cheers
Robert