* Importing from RM - dates from before calendar change

Importing from or exporting to another genealogy program. This is the place to ask.
Post Reply
avatar
Sallion
Gold
Posts: 23
Joined: 05 May 2022 22:21
Family Historian: V7
Location: Oxford, UK

Importing from RM - dates from before calendar change

Post by Sallion » 08 Jun 2022 11:03

I've just noticed something odd with dates from before the calendar change that have two years, eg 1 Jan 1723/4. The dates in FH7 are a year ealier that those in RM7. For example a child who was baptised on 17 Mar 1728/9 in RM7 has a baptism date of 17 Mar 1727/28 in FH7. I have checked multiple dates and it is the same for all of them. I am happy to re-import the data into FH7 as I'm still running both in parallel, but is there any way I can prevent this happening in the future?

User avatar
AdrianBruce
Megastar
Posts: 1535
Joined: 09 Aug 2003 21:02
Family Historian: V7
Location: South Cheshire
Contact:

Re: Importing from RM - dates from before calendar change

Post by AdrianBruce » 08 Jun 2022 11:40

Sallion wrote:
08 Jun 2022 11:03
... dates from before the calendar change that have two years, eg 1 Jan 1723/4. ...
Disconcerting... I only have the free version of RM8 and when I tried to enter a double date like 1728/29, the system abbreviated it to 1728/9. At the risk of upsetting any devotees of that format, that is not valid according to the normal convention for double dating, which requires a two-digit year at the end - i.e. it should read "1728/29". Yes, it is only a convention, but that's what the convention is. So I wonder if the import routine is failing to recognise this as a double date, thinking it's just 1728 (which is actually ambiguous in January) and then treating it as 1728 New Style, i.e. 1727/28 in double dating?
Adrian

User avatar
tatewise
Megastar
Posts: 24622
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Importing from RM - dates from before calendar change

Post by tatewise » 08 Jun 2022 11:51

The double date format is actually more than a convention. It is formally specified in GEDCOM 5.5.1 as:
YEAR_GREG:= {Size=3:7}
[ <NUMBER> | <NUMBER>/<DIGIT><DIGIT> ]
The slash "/" <DIGIT><DIGIT> a year modifier which shows the possible date alternatives for pre1752 date brought about by a changing the beginning of the year from MAR to JAN in the English calendar change of 1752, for example, 15 APR 1699/00.
It sounds like CP should be informed of that odd date conversion on import from RM8.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 08 Jun 2022 14:10

I have traced an entry through export/import, and I think I can see what is happening.

I have an Individual in my FH database who was baptised on 6 Mar 1683/84 (i.e. 6 Mar 1683 in the Julian calendar in use then, but 6 Mar 1684 in the Gregorian calendar).

FH records this correctly as 2 DATE 6 MAR 1683/84 in a GEDCOM export.

RM imports this GEDCOM file correctly, but displays the date as 6 MAR 1683/4, as reported by Adrian.

RM stores dates internally as formatted strings, and viewing this record with an external SQLite viewer shows that it is recorded as D.+16830306/.+00000000.. (where the slash is a flag to indicate a "double date").

When FH does a direct import of this SQLite database, it misinterprets the date, and displays it as 6 MAR 1682/83, but with a sort date of 6 March 1684, which is correct.
Capture.PNG
Capture.PNG (19.56 KiB) Viewed 921 times
If you export the RM database as a GEDCOM file, the date is recorded incorrectly as 2 DATE 6 MAR 1683/4, similar to how it is displayed in the app.

When FH imports this GEDCOM, it fails to recognise the malformed date, and records it simply as a date phrase. If the GEDCOM file is corrected with a text editor prior to import, it imports and displays correctly.

Both apps therefore appear to be at fault - RM is displaying and exporting the date in a non-compliant format, and FH is failing to read the date from the SQLite database correctly, or recognise and correct the GEDCOM formatting.

I suggest that the OP reports this to FH via their support page with a link to this thread so CP can read the discussion and analysis.
Mark Draper

avatar
Sallion
Gold
Posts: 23
Joined: 05 May 2022 22:21
Family Historian: V7
Location: Oxford, UK

Re: Importing from RM - dates from before calendar change

Post by Sallion » 08 Jun 2022 15:02

As advised, I've reported this to CP and sent them a sample file. I'll email them a link to this thread as well.

avatar
Sallion
Gold
Posts: 23
Joined: 05 May 2022 22:21
Family Historian: V7
Location: Oxford, UK

Re: Importing from RM - dates from before calendar change

Post by Sallion » 13 Jun 2022 15:13

Just received a response from CP. They've confirmed this is a problem and they'll let me know when a fix is available.

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 14 Jun 2022 21:32

I'm confident that this fix will be available shortly, as it will be a high priority item given that it corrupts user data. However, it is highly unlikely that CP will provide any method for retrospective corrections (and to be fair to them, that would be difficult to do, as they won't know whether date differences are as a result of their faulty code or a change made by the user after import).

This simple plugin reads your original RM database, and lists all events with dual dates as they are recorded in the RM original. Double click on the individual to see their data in FH. It will therefore be a simple matter to work through the list and make any necessary corrections.

The screen grab shows a reimport of a RM copy of my main project, and the corruption is readily apparent.
The attachment Capture.PNG is no longer available
Attachments
Capture.PNG
Capture.PNG (102.88 KiB) Viewed 729 times
Mark Draper

User avatar
tatewise
Megastar
Posts: 24622
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Importing from RM - dates from before calendar change

Post by tatewise » 15 Jun 2022 09:22

It seems that the Sort Date imports correctly because it is a simple date based on the earlier year of the double Date.
So a Plugin could compare the Sort Date with the associated double Date and fix any discrepancy.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 3774
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Importing from RM - dates from before calendar change

Post by ColeValleyGirl » 15 Jun 2022 09:30

tatewise wrote:
15 Jun 2022 09:22
So a Plugin could compare the Sort Date with the associated double Date and fix any discrepancy.
Assuming that the Sort Date is based on the actual date, and not set by the user to something unrelated to get a specific ordering of facts.

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 15 Jun 2022 10:26

No, FH creates the sort date during import - they were not there in the original RM file. So any automatic solution would have to adjust the event date AND delete the sort date, without corrupting any changes made in FH since importing or sort dates that pre-existed in RM. IMO, that’s way too much potential for further data corruption.

My best guess is that the “typical” user will have only a limited number of such dates (low double figures?) so a one-off manual correction will only take a few minutes. If somebody pops up to say they have hundreds, we can reassess. A bespoke method for them will be easy enough to implement.
Mark Draper

User avatar
tatewise
Megastar
Posts: 24622
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Importing from RM - dates from before calendar change

Post by tatewise » 15 Jun 2022 11:24

FYI:
I have just imported an RM 7 project with a Date: 1 Jan 1750/1 and a Sort date: 1 Jan 1751
The Sort date was set by RM7. I did not enter it and changing the Date automatically changes the Sort date.
After direct import to FH it resulted in the Date: 1 Jan 1749/50 with a Sort Date: 1 Jan 1751
So I don't understand how FH would set that Sort Date automatically from the erroneous double Date.

I manually changed the RM7 Sort date to 1 Feb 1752 and imported to FH again.
That resulted in the same erroneous double Date: 1 Jan 1749/50 but a Sort Date: 1 Feb 1752
So clearly FH imports the Sort Date from RM7 and does not create it automatically during import.

However, facts that have a simple Date do NOT import with any Sort Date, even though they exist in RM7, unless the RM7 Sort date differs from the fact Date, and then the Sort Date does get imported to FH.
Facts with a Range Date do NOT import the RM7 Sort date but FH synthesises a Sort Date for the first date in the Range.
That is probably because the RM7 Sort date is a Range, just like the Date, and FH only allows simple Sort Dates.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 15 Jun 2022 14:51

The key difference between the apps is that RM creates and stores a sort date automatically for EVERY event (although it can be modified or deleted subsequently by the user). The RM sort date for dual-date events is the single Gregorian date, so this is retained because FH thinks it differs from the actual date (arising from FH’s misinterpretation of the RM dual-date).

So yes, it is not created from nowhere by FH, but is still an artefact from the flawed import that would need to be deleted.

There is another oddity as well that I have just noticed. RM creates sort dates even for events that have no actual date. Nothing is shown in the UI, but there is a value stored in the database (and they are all the same 19-digit number). FH appears to import this value, but interpret it as just a string, so it is shown in quotes and only visible under the All tab.
Capture.PNG
Capture.PNG (5.84 KiB) Viewed 631 times
It might be worth raising that with CP so they can fix both issues at the same time.
Mark Draper

User avatar
tatewise
Megastar
Posts: 24622
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Importing from RM - dates from before calendar change

Post by tatewise » 15 Jun 2022 16:12

[EDITED]
I've looked at those odd Sort Dates and they are only on Alternate Name Facts without Dates.
I have found other Facts without Dates that have perfectly valid Sort Dates and none with invalid Sort Dates.

Would it be best for Sallion to add such oddities to the existing dual-date report?

Does the advice on RootsMagic importing need to mention how Sort Dates are handled?
Does that advice belong in https://www.family-historian.co.uk/supp ... gic-import or FHUG KB?
Last edited by tatewise on 15 Jun 2022 16:28, edited 1 time in total.
Reason: Revised regarding Alternate Name Sort Dates
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 15 Jun 2022 17:35

Just because you can't find them doesn't mean they don't exist. How do you explain the screengrab in my last posting?

I created an RM test file with one person and many undated facts. Every fact had an identical sort date in the SQLite database, and importing directly into FH, every fact had a superfluous sort date (even though they are only visible in the All tab).
Capture.PNG
Capture.PNG (13.67 KiB) Viewed 597 times
Mark Draper

User avatar
tatewise
Megastar
Posts: 24622
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Importing from RM - dates from before calendar change

Post by tatewise » 15 Jun 2022 20:23

OK, wierder and wierder! I was only stating what I found and not disputing what you found.

Can you answer my questions.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 15 Jun 2022 22:12

I've created a separate ticket for CP with what I think are the key points:
  1. RM defines a SortDate for ALL Facts and Individual Names, using a common value as the initial default.
  2. This is overwritten by an actual sort date when a date is defined for the Fact or Name (unless that date is just a date phrase). Initially, this sort date is identical to the Fact date, but can be modified by the user.
  3. When FH imports an RM file containing an undated Fact or Alternative Name, it retains these default placeholder values as a text string ("63 6383" in my case, but I don't know if that is universal). Dated names are described in Notes, which appears sensible.
As well as the apparent bug of retaining superfluous placeholder values, I have suggested to CP that both the handling of sort dates and how FH treats dated RM names should be included in their (otherwise fairly comprehensive) overview of the import process. IMO, the KB shouldn't duplicate this, but will need to include it if CP don't want to update their documentation for any reason.

IMO again, RM may be a step ahead of FH in allowing names to be time-bound. We have discussed that feature here before, and perhaps FH is missing out on a useful ability by sticking to GEDCOM.
Mark Draper

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 16 Jun 2022 12:00

CP have identified the root cause of the flawed import of default placeholder sort dates, and it will be fixed in the next minor update. They didn’t comment on updating the documentation,so let’s see what happens. We can add a short note to the KB if they don’t change anything.
Mark Draper

avatar
Mark1834
Megastar
Posts: 1520
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Importing from RM - dates from before calendar change

Post by Mark1834 » 28 Jun 2022 08:07

I have deleted the plugin from the post earlier in this thread as it has now been replaced by the more general Audit RM Direct Import plugin.
Mark Draper

Post Reply