* Making the Move from Generic to Templated Sources

[shoshk]

Use of Title and Short Title

Up to now, we have been storing the fully formatted Footnote in the Title field and a short, descriptive title in the Short Title field, which is formatted such that it groups similar sources together in the Sources List.

For example:
Title:
"England & Wales Births 1837-2006 " database and images, FindMyPast (http://findmypast.com : 14 May 2017), Esau Abelovitz, Q4 1908; citing General Register Office, "England and Wales Civil Registration Indexes," London, England

Short Title
BIRT - ABELOVITZ, Esau (1908) - index (gro)

Our short titles use a system which we adopted during our TMG days.

Keeping the fully formatted Footnote in the Title means that it exports with no problems to other programs.

After making the move to templated sources, I believe that we will want to use the automatically formatted Title field to store what we now store in the Short Title. Within FH, there's no reason to store the Footnote twice. And that will enable us to format the short form automatically.

I was thinking that the export problem could potentially be solved using Mike's wonderful export plugin.

After a preliminary analysis, I believe that the export problem could be solved if an option was added to:

1) export the Footnote as TITL
2) export the Title field as ABBR

Mike, what do you think?

Shosh

[ColeValleyGirl]

I'd welcome this as well.

[tatewise]

Don't hold your breath for a feature like that.

That requires parsing the Source Template record's Footnote Format and interpreting it in terms of Field values including conditional <angle brackets> sections, etc.

What is needed is an API function fhGetSourceFormat(...) similar to fhGetNarrSentence(...) that returns the text derived from a Source record and its Source Template for a chosen Format field. A function like that must exist inside FH.

Even then exporting both TITL and ABBR is not acceptable for the products that do not support ABBR:
e.g. Ancestry, FTM 2019, Family Tree Analyzer, Gramps Family Tree, MyHeritage, TribalPages, etc.
In those cases, the ABBR becomes a labelled Note text.

[shoshk]

It surprises me that CaIico Pie didn't include a function for retrieving the various citation formats. I hadn't thought to check.

So, now I've started thinking...

I think it might be possible to get around this lack by temporarily setting the 'Record Title Format' to the 'Footnote Format' and refreshing it to get the value (this would be used for TITL) and then setting the 'Record Title Format' back to its original value (which would be used for ABBR).

What do you think?

[tatewise]

That is more difficult in the Export Gedcom File plugin than usual because it operates directly on the GEDCOM file instead of through the Project data API functions. That is because there are many 'hidden' tags in the GEDCOM file that cannot be reached via the API functions. I don't have time just now to experiment with your suggestion. Please let me know if you get it to work.

[Mark1834]

I don’t know if this is relevant here or not, but I’ve found Python to be a very effective tool for processing the GEDCOM file directly.

I’ve produced relatively simple scripts in the past for splitting out lumped sources and for my sanitised Ancestry export, which is barely a hundred lines of code.

[shoshk]

Mike, I tested my idea and it works. Here's the code (very simple, no error checking). It expects a templated source.

Code: Select all

function main()

	local tblSource = fhPromptUserForRecordSel('SOUR', 1);
	local pSource = tblSource[1];
	if pSource:IsNull() then return end;

	local pTitle = fhNewItemPtr();
	pTitle:MoveTo(pSource, '~.TITL');
	local sABBR = fhGetValueAsText(pTitle);
	
	local pTitleFormat = fhNewItemPtr();
	pTitleFormat:MoveTo(pSource, '~._SRCT>TITL');
	local sTitleFormat = fhGetValueAsText(pTitleFormat);
		
	local pFootnoteFormat = fhNewItemPtr();
	pFootnoteFormat:MoveTo(pSource, '~._SRCT>FOOT');
	local sFootnoteFormat = fhGetValueAsText(pFootnoteFormat);

	fhSetValueAsText(pTitleFormat, sFootnoteFormat);
	fhSrcEnableAutoTitle(pSource, true);
	local sTITL = fhGetValueAsText(pTitle);
	
	fhSetValueAsText(pTitleFormat, sTitleFormat);
	fhSrcEnableAutoTitle(pSource, true);
	local sOriginalTITL = fhGetValueAsText(pTitle);

	debug = debug

end

[shoshk]

Mark, I wouldn't mind getting a copy of your Ancestry export, if you're willing to share.

Thanks,
Shosh

[Mark1834]

Shosh, this is an updated version of the script discussed here. In essence, it exports a highly sanitised copy of my database (main facts only, no sources or media, and all reference to living or private individuals or their families removed completely). This purging goes further than most other similar options, and I find it useful for protecting the privacy of recently bereaved individuals or where there may be family sensitivities around matters that are not fully public.

This is imported into RootsMagic, and is used to update my master extract there, which is then subsequently uploaded to Ancestry to properly exploit their hints system and public visibility.

99% of what it does is simply selecting the relevant lines from the GEDCOM file to copy, with some minor reformatting to cope with RootsMagic not liking missing names and Ancestry recording Census as Residence. You could undoubtedly do all this with a FH plugin, but at the time of producing this my Python was stronger than my Lua!

This forum does not permit attaching .py files, so I have simply appended .txt to the filename (it's not very bright... )

FH2Ancestry106.py.txt

(4.6 KiB) Downloaded 118 times

[shoshk]

Mark,

Thank you.

I decided to implement the export as a plugin and your file was a good starting point. I've never programmed in Python and although I can read and mostly understand your code, I prefer to work in lua, which is slowly becoming more familiar.

I've got the plugin working and now I'm playing with seeing how updates in FH are reflected in RootsMagic after a new merge.

I'm curious about what your experiences have been.

I've already figured out that deletions in FH have to be done manually in RM.

I'm sure there are other changes that can't be automated.

Shosh

[Mark1834]

Shosh,

I summarised my experience recently here. The only change is that I now produce my names of individuals to review via a simple plugin rather than a query, as it is automated to generate an ordered list of all individuals updated since the date/time stamp of the latest RM import. That means I don't have to enter the date each time, and I can base the comparison on full date/time so catch anybody updated immediately after the import.

_RootsMagic Sync List.fh_lua

(2.17 KiB) Downloaded 101 times

I would say that the biggest limitation is having to rebuild families manually in RM. I did an update yesterday, and I had probably left it a bit too long as I had about 20 brand new individuals spread over several families. It would have been better updating each family individually.

[shoshk]

Mark,

Are you using RootsMagic's facility of assigning a unique identifier to each individual to help automate the merge process?

[shoshk]

Another question...

In the other post, you mention that there is a problem transferring new relationships to RM.

I don't think I'm seeing that behavior. I added parents and a sibling (new individuals) to an existing individual. It seems to have transferred to RM with no problems.

Can you be more specific regarding your problem transferring relationships?

[Mark1834]

The imported GEDCOM file is fine in RM - it is bog-standard GEDCOM that is read without any problems. My problem comes when I compare the new import with my master file that is synced with Ancestry.

In the two screen dumps below, I created a new individual in FH, re-ran the export and imported into RM. Comparing databases correctly identifies the new addition, but when I copy it across the family relationship is lost, and I have to manually reassign which family they belong to (add child - existing record, similar to FH).

Is there a setting to copy relationships as well that I have missed?

Capture1.PNG (20.44 KiB) Viewed 4833 times

Capture.PNG (20.99 KiB) Viewed 4833 times

[tatewise]

@shoshk, the Plugin script you posted yesterday is looking good and may provide the Title & Short Title formats requested.

[shoshk]

Mark, I use a different technique for fh -> rm

RM assigns a unique identifier to each individual. For simplicity's sake I won't go into the particulars of how I do it, just know that I transfer the _UID for each individual from RM to FH and it is included in the tags I export.

I import the entire exported gedcom into my RM database. At this point, everybody in the database is duplicated. Then I run an automatic merge which uses the unique identifier to match up individuals with absolute reliability.

It seems that my technique preserves the relationships. I have no idea why.

[shoshk]

I decided to start a new thread for discussing FH/RootsMagic/Ancestry. You can find it here: FH -> RootsMagic -> Ancestry (Work-in-Progress) (18572)

I appreciate your input.

[tatewise]

There is an updated Export Gedcom File V4.1.2 (18578) that incorporates the Templated Source reformatting of Title (TITL) and Short Title (ABBR) fields discussed earlier.
To invoke that mode for any target product, open the Extra Options tab, and for Source Template 1 _SRCT: lower-left select Reformat Metafields. It uses the Footnote Format: template for the Title and the Record Title Format: template for the Short Title. It would be feasible to use the other two templates to set other generic Source fields such as Publication Info, etc.

[shoshk]

Mike, you are a star!

[tatewise]

Shosh, without your brilliant scripted suggestion, I would not have even attempted to start thinking about it.

Does the option Reformat Metafields convey the correct meaning or is there a better phrase that is no longer?

[shoshk]

Mike,

Ah, how I love to hear that I've made a brilliant suggestion!

Using 'Reformat Metafields' absolutely requires that the user will have to read the help in order to understand what it means. It would be nice to find a more description phrase, but I can't think of one off of the top of my head.

I'll stick it in the 'mulling it over' section of my brain. Perhaps something will pop into my head as I'm waking up one morning.