Family Historian User Group

Your work is much appreciated Mike. I have downloaded and will get back to you.
I have four of my personal Projects in fh - and all still exist in TNG so I can output gedcoms with sort dates. But once I have run the plug in I have little idea what I am looking for.
At a very basic level my concern with the sort date problem arose from the possibility that my Events might be out of order in a re-export to Gedsite. For which I would be using your other plug in. But one step at a time.

After running this Plugin in a COPY of your Project you can open and expand the All tab of any Individual or Family record in FH.
By expand I mean double-click on the Name under the heading Record until there are no [+] icons, only [-] icons.
Then every Fact should have a Sort Date and those that matched with the TMG GEDCOM should have the TMG _SDATE Sort Date value

That post has excellent clear instructions Mike. Thank you. I managed to follow them but would not have been able without your clarity.
I have not done a full test as had other things happen today. Just one tree so far.

I made a copy of my grandmother's tree 1689 people in 652 families which I have had in fh for some time and on which I have added further people.
I went to TMG and exported a new gedcom with sort dates from the copy there 1615 people.
I ran your plug in opting to use that GEDCOM file.
(I got no message that the plug in had run - but assumed it had)
I looked at the copy fh project.
I then opened the record at random of someone near the start of the alphabet and on the All tab removed all the plus signs. Each event now has a sort date. As you said, her death for which I just have the year 1959 is given a sort date on the last day of that year 31 December.

I then opened the record of my great uncle for whom I also have only a year of birth and the same applies. I looked at a person I added only after moving to fh. And he has sort dates there too. Even tho he is not on the Gedcom.

That convention for sort date being the last day of the month or year looks odd only when I have entered a period between, or after a certain date. But as there is a genuine date in the date field, the sort date doesnt apppear in the reports. It just creates a report that makes good sense. I generated a couple of reports and they look good.

There seem to be sort dates added to all events including the undated ones I put in for my Gt Grandfather's career. And his biographical record on the All tab looks much as it did in TMG.

There are no sort dates on my working copy of the project on which the plug in had not been run.

Small question - If I subsequently add the day and month of birth to one of these records ( probably from the 1939 register) will that override the sort date?

Is there anything particular you would like me to look for? On this or another tree?
But your plug in orders things so well, I will apply it to all the projects I imported from TMG. Waiting only till you have finalised.

Mike - as indicated, I have carried on testing the plugin output.

This time, after creating a new copy of my main project (so there were no Sort Dates), I added a number of Sort Dates manually using the LMO approach you originally outlined - these were primarily on Birth and Baptism facts where I had a full date for one or the other fact, and only a Year for the other (e.g. Birth = "1750" and Baptism = "12 June 1750"). I undertook a similar exercise on a number of Death and Burial facts. I then ran the new Plugin and reviewed output - I could find no issues other than the two points I raised earlier on the records I checked. Those facts where I had manually created the Sort Dates retained my input (as expected).

For info, I took things a bit further and subsequently ran the "Export Gedcom File" plugin against this project and then utilised Gedsite to build my website (I find it's sometimes easier/quicker to look at Fact data using my website rather than FH) - again, all looks fine.

As before, I'll carry on running various tests and let you know if I encounter any issues.

Evelyn, thank you that excellent feedback.

I am working on providing better feedback messages after the Plugin has run to say what it has done.

When only a partial Date is provided by the database, I decided the Plugin should create an explicit Sort Date, and chose to use last day of month for a missing Day, and December for a missing Month. Although, there has been some discussion on whether that is always the best policy, and whether say the 1st and January would be better.

The main thing to check is are TMG _SDATE Sort Dates being transferred to FH Sort Dates particularly for undated Facts?

The question about what happens if a Fact actual Date is changed is one I had not considered.
Ideally the user should change the Sort Date via the All tab at the same, but that can easily be overlooked.
My first reaction is that the Plugin should report such inconsistencies between Date and Sort Date in the Result Set and leave the user to make the desired adjustment.

Brent, when this Plugin is finalised I intend to update the Export Gedcom File Plugin to use LMO Sort Dates where they exist instead of synthesising Sort Dates as it does now.

Within the next few days Mike, I hope to use the plug in (utilising the TMG gedcom) with the other three family trees I brought in to fh.

I guess you have given me a feeling of security.

I have two other trees still in TMG - taken back there really because I worried too much about data loss to trust the fh copy. On one of those I recorded the voyages of a great uncle who travelled from UK to Australia and back by various routes almost every year. His record of those voyages lacks precise dates. Tho many Are on the passenger lists now. I ordered them in TMG and I am really looking forward to seeing what they look like after running your plug in. But it will involve bringing the Project into fh first. I apologise that this may take me some time. I am out all today.

About adding dates subsequently. I had envisaged using your plug in with the TMG GEDCOM option only once ever on each project - as part of the import routine.
After that the Project could be handled as a normal fh Project?
So I had imagined, e.g. before export to Gedsite, or if one wanted to order events for a report output, one would do as jbptascott has done and run it without referring back to the old TMG gedcom?

But may be I am confused about this? Or the additional uses of the plug in.

Evelyn - Unless I have missed something, Mike indicated that the new plugin can be run multiple times as one of the criteria it uses is "Does a Sort Date exist on the Fact record?", and if it does then the plugin skips that Fact.

I plan on running the "final" version of the plugin against my FH project and will then only occasionally utilise it after that (probably before I run a new Export for GedSite). It is possible to add the Sort Date to Facts manually (using the LMO process Mike indicated) but I suspect that I will normally forget to do this particularly when using AS to capture data!. If Mike adds something that lists all "new" Sort Dates that have been added during each execution of the plugin, then it will be easy to review / change the few records that have been modified since the plugins last execution.

Yes, the Plugin works in various modes automatically.

If there are fewer than 9 existing LMO Sort Dates then it assumes initialisation mode, and provides the prompt to choose the GEDCOM File. In the next version it will list a summary of how the LMO Sort Dates were derived. This mode will usually only happen once.

If more than 9 existing LMO Sort Dates then it assumes normal mode and there is no prompt.
It lists a Result Set for any missing LMO Sort Dates and how they have been corrected.
The next version will also report any discrepancies between actual fact Date and LMO Sort Date.

In both modes it sorts facts into LMO Sort Date order.

Here attached is next version of Order Facts by Sort Date Plugin Version 0.4 date 11 Dec 2017.
[ Attachment deleted as superseded by many later versions. ]

It adds the following new features in response to earlier feedback:

1. When Fewer than 9 Fact Sort Dates detected it prompts the user as before.
   If Help & Advice is used then a browser help page is displayed, but is still under development.
   If Apply Dates is used then all missing Sort Dates are derived & sorted as before, but now there is a closing summary of how those dates were derived.
2. A Fact Before date with blank day or month defaults to 1st & January and 1 day subtracted before setting Sort Date.
3. If there are more than 5000 Individual & Family records, then a Progress Bar indicates progress.
4. When 9 or more Fact Sort Dates detected, a final Result Set lists every newly assigned Sort Date and those that differ by more than 9 days from their Fact Date. That caters for brand new Facts, and old Facts whose Date has changed, but their Sort Date has not been updated.

Remember that this is still a prototype Plugin, and has not yet been exhaustively tested, so use it with caution.

Mike - looking good so far!.

Your b & c both work as described. The new "results" Record Set is very helpful.

A few things I've noticed so far (NB the plugin was run against a new copy of my main project)

Issues:
I was unable to access a Help button regardless of how many times I ran the plugin (at all times my project had more than 9 Facts without Sort Dates!). I imagine users will need this when they run a refresh some months after their first "all encompassing" run of the plugin.

Other Issues (possibly more for inclusion in the Help than requiring code change):
#1 If a Fact Type has no Abbreviation defined (it is not mandatory in FH) then the Record Set "Fact Summary" is blank;

#2 With an Attribute Fact Type of "To Do", the "Fact Summary" shows the Value set against the Fact. I have a number of these where the Value is "Census", "Burial", "Birth", etc (indicating that I need to review / find these pieces of information, etc). However, there are also valid "Census" Facts, so the "Fact Summary" can report the same word ("Census") for both a To Do / Census and a "real" Census fact.

I'll carry on with more tests tomorrow and report back on anything I find, but as I said, everything so far is looking good.

The Help & Advice button only appears in the initialising mode dialogue when Fewer than 9 Fact Sort Dates detected.
When published in Plugin Store, there it will say "See Order Facts by Sort Date Plugin for Help & Advice."

#1 Please supply more details as the Abbreviation should have no impact.
Although, I suspect it is simply another manifestation of #2 where an Attribute has no Value.

#2 This is an annoying feature of FH, but now you have mentioned it, that is easily fixed in next version Sorry

Thanks Mike

In respect of the " If a Fact Type has no Abbreviation defined (it is not mandatory in FH) then the Record Set "Fact Summary" is blank", I have attached some screen shots of the Memorial Fact for Selina Tapscott:
Based on your responses, I have checked further and can confirm that the blank "Fact Summary" only occurs when the Fact Type = "Attribute" AND the Fact has no Value input - viz. a Fact Type of "Attribute" which has a blank Abbreviation in the Fact Definition but which has a Value on the Fact reports correctly and a Fact Type of "Event" which has a blank Abbreviation in the Fact Definition reports correctly.

I have also noticed that in instances where the Fact Type is "Attribute" and the Fact has a value against it, then the "Fact Summary" only reports that value (viz. it does NOT contain the Fact Name) - see example below (the "5" and "Servant"):

Yes Brent, that is a 'standard' FH characteristic.
Try using View > Standard Queries > All Facts and examine the Event/Attribute column.
It exhibits exactly the same characteristics regarding Events versus Attributes.
I had overlooked that, but is already fixed in the next Plugin version, pending any other anomalies that need fixing.

I've been going through Family records, including Marriage Facts with no data and those with no Date, and all have had the Sort Date added in line with the "rules" you originally outlined. Further checks of Individuals Facts showing no additional issues.

A couple of general things I have noticed:

#1 If an Individual has multiple Marriages and no dates are present on any of them, then the Sort Date on each Marriage Fact is set to "1 January 3000" on each. (Probably something that is more for the Help than requiring code change)

#2 You changed code such that "A Fact Before date with blank day or month defaults to 1st & January and 1 day subtracted before setting Sort Date" - is there a case (for consistency purposes) for making a change for an AFT date such that it defaults to 31st & December and then 1 day is added before setting the Sort Date?

#1 Yes, Sort Dates on Family facts such as Marriage are a bit of a problem, especially if undated.
It can be worse with multiple Marriages if some are undated and some have Dates.
Undated ones with Sort Date of 1 Jan 3000 will come after those with a Date even if the Family link is earlier.
The impact may not be so obvious in FH but will affect products such as GedSite more.

I have derived most of the tactics in this Plugin from the Export Gedcom File Plugin.
But some revisions are being applied such as the Before Date fix.
For technical reasons, the Export Gedcom File Plugin works through the GEDCOM file rather than the Project, so its options are restricted, whereas this Plugin can interrogate the Project database more easily, and may be able to make more rational decisions about the Sort Dates needed for undated Family facts. I will have a look.

#2 That adjustment for AFTer Dates sounds reasonable.

Following my question about undated marriages in General Usage.
In TMG where marriage Events were created - My sort dates for undated marriages drew on birth or baptism dates of the children - I tended to give a marriage date bef. the date of birth of the first child.
I do realise that in some cases unwed parents married many years later - but as a general rule I liked to output
Marriage 1.
children of that marriage,
followed by marriage 2
and then children of the second marriage.
As these were only sort dates there were no qualms about accuracy.

Mike - please let me know if you would like me to trial these new versions and on what?
Or trial and then re export using your Gedsite plug in?

Let me post the next version that has some significant improvements for Family events such as Marriage.
Also the Export Gedcom File Plugin needs update to use these new LMO Sort Dates.
Should be ready in a few days.

Thanks Mike. Will wait then. As you know, this is so good that it is a temptation to run it for all my projects.

I have two attached updated Plugins for you to try.

Order Facts by Sort Date Plugin Version 0.5 Date 12 Dec 2017.
[ Attachment deleted as superseded by many later versions. ]
This has the following new features:

1. Mostly complete Help & Advice page at plugins:help:order_facts_by_sort_date:order_facts_by_sort_date|> Order Facts by Sort Date Plugin
2. Tidier summary of derived Sort Dates after first run when Fewer than 9 Fact Sort Dates
3. Fix Result Set display of Fact Summary column for Attribute facts
4. After Dates now add 1 day to Date to derive Sort Date
5. Instead of 1 Jan 3000 the initial synthetic date for undated Family facts is derived from the estimated Birth Date of any Children, or the estimated Death Date of the Spouses. That should ensure such facts are in the correct order when exported to such as GedSite, but may not help much in FH.

Export Gedcom File Plugin Version 3.4.1 Date 12 Dec 2017 ZIP file that must be extracted.
This now recognises LMO Sort Dates created by the above Plugin.
For such as GedSite and RootsMagic it adds matching _SDATE Sort Dates.
Except for GedSite and FH V5, it removes the LMO Sort Dates from the exported GEDCOM.
I am not sure whether they should also be removed from GedSite exported GEDCOM.
[ Attachment deleted as now published in Plugin Store in Feb 2018, and removes LMO Sort Dates unconditionally. ]

When any _SDATE Sort Date is derived from a fact Date (i.e. no LMO Sort Date) it now uses the same Before and After algorithm as Order Facts by Sort Date.

Thanks Mike - have downloaded these and will test over the next few days.

All looking good so far Mike. As intimated earlier, I am not using a TMG file - I also have never utilised the Note labelled "Sort Date: " approach so have carried out no testing involving these (your points 2 and 3 in the Help "Sort Date Auto-Derivation" section).

In respect of point 5 "Undated facts, or same Date as previous fact, use the previous fact Sort Date + 1 day", I have found one slight anomaly that I am recording here "for the record" (it actually had no impact on the data on my website, but may impact others who have utilised the Witness functionality).

In my case, I input details from a Birth Certificate via AS, including Name, Occupation and Residence of the Informant. This resulted, in FH, in a Witness / Informant on the child's "Birth" Fact and two Facts ("Occupation" and "Residence") against the Informant - all three Facts having the same date of "9 October 1919". After running the plugin, the "Witness / Informant" and "Occupation" had LMO/Sort Dates of 9 October 1919, while the "Residence" fact had a LMO/Sort Date of 10 October 1919. The same Sort Dates were retained when the "Export.." plugin was invoked and the data imported to GedSite. For something like this I am not overly worried about the final order of the three facts when reported on my website, but it is possible that it might have more serious issues when used with other Facts (I can't think of any off-hand, but others might!).

I am not entirely clear about which facts belong to which Individual.
Nevertheless, the point is that there needs to be a way to determine the desired order of identically dated facts.
That is, after all, the primary purpose of Sort Dates, otherwise what are they for?
So the Plugin automatically gives successive Sort Dates to facts with identical Date values, whether actual or blank.

If the user needs a different order then the Sort Dates must be manually changed.
That was the purpose of the now obsolete Note labelled Sort Date: format.
Then not only will the Plugin adjust the order of the facts in FH but so will products like GedSite when supplied with an exported Gedcom.

tatewise wrote:Nevertheless, the point is that there needs to be a way to determine the desired order of identically dated facts.
That is, after all, the primary purpose of Sort Dates, otherwise what are they for?
.

Well, no Mike. The primary use of Sort Dates is to maintain the order of Events when the date is unknown or only partially known.

Or to quote the TMG Help:
Sort dates are used to maintain chronological order on the Person View, and in narrative reports, Individual Detail reports and Family Group Sheets. They do not print or export and have no genealogical significance.

Don