* Apparent corruption of Saved Charts

[MarionWoolgar]

I am currently using Family Historian version 6.2.7 & operating system Windows 10 build 19042.

On 17 Jan 2015, I created a Chart for Mary Ann Hopestill HERSEE and it hasn't been changed since that date. Yesterday, I wanted to reproduce a print of that Chart in connection with a new piece of research that I am planning to do this year when record offices reopen. What I found when I opened this Saved Chart in the the file attached as printscreen1 and I was appalled by what I saw.

Fortunately, I have a large collection of back-up drives and I also never dispose of a computer without removing the HDD. I have therefore been able to find a copy of my Family Historian data files that were last used on 27 Nov 2017. I imported this old Saved Chart & used it to reproduce the original Chart that I saved, as shown in the file printscreen2. The difference is obvious.

Can anyone please explain what on earth has happened to this Saved Chart, please? I have been using Family Historian since Version 1 & I have produced a very large number of Charts in that time. If I have to review them all to see which are OK & which need to be restored from my pre-2017 Saved Charts, well, that is going to take up a lot of valuable time.

All suggestions gratefully received,

Marion Woolgar

[tatewise]

The usual reason for Saved Charts becoming upset is some change to the Record id of the Individual records involved, which are the unique keys that the chart uses rather than record names that are not unique.
That happens when records are renumbered via the Tools > Record Identifiers... command or are merged into another Project via the File > Merge/Compare File... command.

However, the fact that an old copy of the Saved Chart imports to the same Project and works correctly suggests the Record Id have not changed, so no obvious explanation comes to mind.

It is difficult to tell, but do the trio in printscreen1 appear in printscreen2?
What are the Record Id of that trio? Does their Property Box match the Chart?

[MarionWoolgar]

Well, you have given me a lot to think about there, Mike.

Firstly, I used to hold my HEARSEY data in a another FH Project, but when I discovered that this particular branch of the family was connected by marriage into my WOOLGAR ONS, I split out the HEARSEY data from its original project & merged it with my WOOLGAR ONS Project. That is the only time that I have ever used File, Merge/Compare File and that happened several weeks before I produced the original Chart in 2015. I then applied to the combined file the extra data that I had found in Probates & MI's that proved the connection between the two families and produced the original Saved Chart. This was a major & totally unexpected advance in my ONS that solved so many long-standing problems, and that's why I remember the sequence of events so clearly. After all, it isn't every day that you discover a stained-glass window with an inscription relevant to your research in Chichester Cathedral! Since then, I haven't added any extra data to this branch of my husband's family tree, nor have I had a need to re-examine the Saved Chart until a new researcher contacted me earlier this week & I needed to open the Saved Chart to refresh my memory.

However, I do quite often use Edit, Merge/Compare Records and that would certainly delete some Record ID's. Do deleted Record ID's get recycled?

According to my current database, the Record ID's of the trio are: Unknown WOOLGAR = 19698; Lizzie Unknown =19699; Walter J. WOOLGAR =19700. That data was entered from a 1901 census entry of a widow and her only child & was entered on 21 Jan 2015. As per printscreen1. My Saved Chart for Mary Frances Hopestill HERSEE was created on 17 Jan 2015 i.e. 4 days previously.

In my old database, those same Record ID's were recorded as: 19698 = Mary Salter ROBERTS; 19699 = Mary Frances Hopestill HERSEE; 19700 Richard RUGG. As per printscreen2.

In my current database Mary Salter ROBERTS = 19309; Mary Frances Hopestill HERSEE = 19310; and Richard RUGG = 19311.

Clearly, the Record ID's have changed, but I cannot account for how, or why, that would have happened. Curiouser & curioser!

Marion Woolgar

[tatewise]

Deleted Record Id are not recycled, and you will typically find gaps in the numbers.

The fact that the same Record Id refer to the two sets of Individual records that appear in the two Saved Charts suggests it is a Record Id problem.

Certainly, merging records will change the Record Id of the Individual records that get deleted as a result of the merge.
So could that explain why the Record Id of the three key Individuals have got smaller?

It is less clear how those Record Id that were in use in the old Project have been reassigned in the current Project.

What I don't understand is how the old Save Chart with the old Record Id picks up the records with different Record Id in the current Project. Or are you opening the old Saved Chart in the old Project?

[MarionWoolgar]

Oh, yes, I did open the old Saved Chart in the old project.

To get to it, I had to utilize the HDD that was originally installed in an old laptop. I fitted the old drive into an external housing, opened Windows Explorer and navigated to the GEDCOM Project file in the old drive. That opened in the copy of Family Historian on that old drive. Once I had confirmed that the old Chart did exist on the old drive, I shut that copy of Family Historian down.

I then opened my current copy of Family Historian on the HDD of my current laptop and imported the old Chart from the old HDD into the current Project File and allowed that to overwrite the old Chart that showed just the trio. I then crossed my fingers and opened the imported Chart & it worked.

[MarionWoolgar]

I have just sorted my current project data into Record ID order. To my astonishment, there isn't a single gap in the sequence of 29090 Individual Records. I must have Merged/Compared Records hundreds of times in the past couple of months alone, and goodness knows how many thousands of times over the years, as I have been working through my database merging records because new information has been identified. At no point have I manually activated Tools, Record Identifiers to generate a renumbering of the sequence of Record ID's. And yet it has obviously happened. How can this be?

Marion

[LornaCraig]

That is extraordinary, and I can't think how it could have happed if you have never used Tools>Record Identifiers. Perhaps Mike will manage to think of an explanation. However you say you have done many record merges in the last couple of months alone, which suggests that the renumbering must have happened very recently, since the most recent time you did a merge.

If you have backups made during the last couple of months can you discover when the change occurred? If it was very recent it might be easier to revert to that version and re-do your latest data entries rather than re-work all your saved charts.

You do need to be aware that even without the unexplained renumbering of all the records there are potential problems if you merge records after creating a saved chart which uses some of those records. When you merge two records one of the record ID numbers will cease to exist, and if that was the one used in a chart the chart will no longer work.

[tatewise]

Marion, please confirm that the largest Individual Record Id shown in the Records Window is 29090 and that the number of Individuals: listed on the lower right border of the main FH window is also 29090.
If so, as Lorna says, that is extraordinary given the history of merging and new data you describe.

As an experiment, if you temporarily add a new Individual record does its Record Id equal 29091?
If so, and the first question above was true, then that also confirms records have been renumbered.
Use Edit > Undo to clear that temporary addition.

Apart from Tools > Record Identifiers..., there is no other way Record Id get renumbered that I can think of.
Not even Plugins can do it.
The Tools > Record Identifiers... has a Caution that says saved charts, books and Internet Data Matching may have problems.

I agree with Lorna that renumbering has occurred after your last Individual record Merge (other records don't matter).
Do you know when that Merge occurred?
What Project Backups do you have dated around that time and since?
Review File > Backup/Restore > Revert to Snapshot... and check if any Snapshots Taken were around that time.
Tell us what you have available as backups and snapshots.

[MarionWoolgar]

Dear Lorna & Mike,

It is very kind of you to help me with this problem.

I have been through my various laptops & back-up drives. First, I must explain. When I change a laptop or put in a new HDD, I never throw away the old hard drive, I keep them just in case of trouble. I have two laptops, one that I am using on a daily basis (HP1) and one that I keep for regular backups (HP2) & take to record offices. In addition, my husband has a very old Samsung on which I occasionally save a copy of my FH Projects, just for safety. In addition, I have 5TB external drive that I back-up everything to once a week - that was done on Friday, so is no use for this purpose; plus I have a 500GB external drive that is just used for my Woolgar ONS that is backed-up monthly. I also have three old, former HDD's that I have kept just as they were the day that I unscrewed them from their laptops. I'm sure all this must seem haphazard to you, but it works for me - or it has, until now! In my view, there is no such thing in life as too many backup drives!

I am attaching an Excel file showing some data from each of the above resources. I have also experimented with each separate HDD as to whether I can view the test file that I used in the two screen prints. As you will see, this problem of mine may go back some way. The last time that the GEDCOM file (according to the available backups) was capable of correctly producing the Chart was 31 Dec 2018. I do add to my collection of Saved Charts from time to time, but rarely go back to them, so I suppose it is possible that the problem could have been in hiding for that long?

I did once upon a time consider renumbering, saw the caution & didn't do it.

Yes, I did Add Unrelated Individual & a new 29091 was created. I did it again & 29092 was created. I actually did that late last night & stupidly deleted them afterwards, rather than using Edit, Undo - as I should have done. However, please note that the spreadsheet shows that I haven't had the two Record ID totals matching in a very long time.

I have been using Family Historian since 2003 & in the intervening years, I have been loading 30 years of research into it. Is there any way of recovering the situation, do you think?

Marion

[tatewise]

Firstly, your statement "To my astonishment, there isn't a single gap in the sequence of 29090 Individual Records" is mistaken as the spreadsheet shows consistently that the number of Individual records is less than the highest Record Id so there must be some gaps. In the latest HP1, there must be 29090-28750=340 missing Record id gaps but they will be difficult to spot by eye.
That means they have NOT been renumbered.

It seems that something upset the Record Id of the Individuals in the Saved Chart between 31-Dec-18 and 31-Oct-20.
Only by comparing those two versions of the Project might you discover how the Record Id changed.
I.e. What Individual records have the same Record Id in the earlier and later Projects for these key Individuals.

Nevertheless, it is not clear what may be the best way to resolve things without regenerating the Saved Charts.
If somehow the current Record Id could be restored to the earlier values, would that upset Saved Charts produced more recently, and if so how many?
I'm thinking that perhaps the latest Project could be Merged into the older Project, which would retain all the older Project Record Id but all records added since then may get new Record Id. That way the old Saved Charts should work but more recent Saved Charts may not. This is a novel situation that I don't ever recall before.

[LornaCraig]

It’s reassuring to know that there has not been a complete renumbering because that would have been impossible to explain.

Most of those missing 340 Record IDs can probably be accounted for as the result of merging records. However it is still not clear how the ‘trio’ of people came to have the same ID numbers (19698, 19699 and 19700) in your current database as three completely different people in your old database. Deleted record IDs are not re-used when you record new people, so I am wondering whether the ‘trio’ were merged in from another project in which they happened to have those numbers, and because the numbers were not already in use in the main project they were able to retain the ID numbers from their previous project.

Like Mike, I had been wondering about merging the current project into the old one to retain the older IDs but as you probably know, merging projects is a fiddly business. You will need to weigh up the effort of the merge with the potential benefits.

Do you have any idea how many of your existing charts no longer work? Are most of them still OK, or are most of them displaying problems? And to get some idea of how many newer charts might be upset if you restored old record IDs, look at the Charts folder inside your project and sort the list by Date Modified. That will show you how many charts have been produced since Dec 2018.

I know this won’t help now, but whenever I save a diagram as a Family Historian Chart I also save it as a PDF. The PDF is not dynamic of course, so it won’t reflect any additions to the data, but it does preserve a version for reference.

[paultt]

Please be aware that there is a 'bug' in v7.0.3.0 which I have reported in support ref #469349 about the saving of charts, particularly the All Rels. What I found was that I could open up an all relatives chart and everything looks perfect. Now I saved the diagram as a F H Chart. Keep this original chart open, and load the saved chart you just created so you can toggle between the two. The saved chart, either in the saving, or the redisplay, has random sections where the children and siblings may have re-ordered, sometimes overlapping the vertical drop lines, and leaving extended horizontal line to where the re-ordered sibling should have been. I have been able to reproduce this quite a few times, but never thought of checking whether a save as pdf saved correctly. If it does, then most likely the error is caused by the reopening and display of the save chart.

[tatewise]

I have reviewed Marion's Saved Charts that were posted on Saturday and wonder why they need to be Saved Charts.
Unless I'm mistakem, there appears to be nothing in the chart that needs any more than a Custom Diagram Type would provide, apart from the Diagram Root. The things that a Chart saves are repositioned boxes and bars, inserted diagram branches, customisation of selected boxes, etc, but I don't see that in the posted samples.
So rather than recreate all those Saved Charts why not just build the required Diagram as and when required?

Bear in mind that Saved Charts are NOT 'frozen' in time. Any new Individuals added to the Project can add new branches to a Saved Chart, and any new Facts added to Individuals will update the text in boxes in a Saved Chart. If you need 'frozen' charts then only a save to PDF will achieve that. Otherwise, choose the desired Custom Diagram Type, select the Diagram Root Person/Couple and the required chart will be produced based on the current Project data.

[LornaCraig]

Yes, I think you are right about that particular chart Mike. It doesn't appear to have had any particular branches moved or closed. However in another thread Moving box within diagram (19174) Marion was working on a diagram where some branches were hidden. I suspect that she may have a mixture of charts where some are more customised than others.

Marion, presumably you already have a custom diagram type which you use whenever you want to create and save a new chart? (The ones we have seen all have red text for the names, and no boxes.) So re-creating your charts when you need them may not involve as much work as you think it will. The only slightly time-consuming ones will be those where you have closed or moved selected branches or perhaps inserted text boxes.

[MarionWoolgar]

Thank you all very much for your advice.

Re: Paultt's contribution on the subject of V7 bugs, I did purchase it and installed it on one of my laptops (HP1). Several problems arose with citations etc and I might have been able to deal with those, in time. However, when I saw that my own sister had become disconnected from the rest of the family & treated as an "orphan" record, I uninstalled it. I then reinstalled v. 6.2 and imported my saved project files from my backup files on HP2 and I have been using that ever since. When & if the initial problems with V7 have been resolved, I'll probably have another try with it.

I don't think that it would be a good idea to try something tricky with my database - I might make a terrible mistake. I now know that I can go to one of my back-up project files and recover an old Saved file, if I need to, and meanwhile I have marked all those old drives to make sure that they are not overwritten for any reason.

If I prepare a Chart to give to someone, I usually do make a*.pdf copy and file it to the appropriate correspondence file. In the case of the HERSEE Chart that got scrambled, the data behind it was compiled at the end of 2014 into early 2015 as a result of an extraordinary find of an MI that I followed up with lots of research & I combined three branches of the WOOLGAR & HERSEE families together and made a series of discoveries. I believed that line to be "dead" at both ends, until I received an extra piece of data last week. I am confident that nothing has been changed in that ancestral line since I competed it in 2015. Clearly some renumbering has taken place somewhere along the line and that is obviously going to remain unexplained.

to be continued

[MarionWoolgar]

continued - sorry I pressed the wrong button! My hands aren't as good as they used to be...

Yes, Lorna is right, I have customized my Charts and I first did that some years ago. I dislike those little coloured boxes, so I cancelled them and also set up my own text scheme with particular fonts & colours so that names are more prominent than data and also to get away sometimes from ubiquitous fonts like Arial & Times Roman. I also add text boxes as a heading in the top left and another text box in bottom right to add my name, E-mail address & the date. I think that they are an improvement on the defaults. I don't often use All Relatives Charts, because they tend to be huge & unwieldy; but sometimes, I have a need for the extra flexibility that they offer. Sometimes, I also need to add little trees into bigger lines and I find that pretty tricky too. Having struggled to create them, I usually do file them as a Saved Chart. The Chart that was causing me problems in the thread Moving box within diagram (19174), is a case in point because I wanted to concentrate on the direct male line in connection with some Y-DNA results.

I am relieved that my data hasn't been corrupted or lost, so thank you all very much for that reassurance. I have recently been concentrating on tidying-up my database by locating "orphaned" records and using Edit, Merge/Compare to link them to the correct family. i am waiting to go into hospital for an operation & I won't be able to do much research for a while afterwards, so I want to try and leave things as tidy as possible until I can pick up the threads again.

[paultt]

MarionWoolgar
As an update to my previous post on corruption in saved charts, I have been notified by Calico Pie that the bug has been resolved and the fix will be included in the next point release, probably 7.0.4. No date yet on the availability, but should be soon.

[MarionWoolgar]

Thanks very much for that information, I will watch out for the update in due course.