* Batch updates to Date of Death?

Writing and using plugins for Version 5 and above.
Post Reply
avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Batch updates to Date of Death?

Post by Sharon » 28 Aug 2019 20:11

I'm looking for some means - possible a plug-in or maybe a query - to change 'Date of Death', where it currently contains the word 'circa' plus a year to contain only the phrase 'n/k' . I tried the Search and Replace plugin, but it will only replace the actual word 'circa' and leaves the year untouched - unless there is a setting I haven't found which will just overwrite the whole field with 'n/k'.

Does anyone know if there is a way of achieving this please?
Thank you,
Sharon

User avatar
tatewise
Megastar
Posts: 16469
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Batch updates to Date of Death?

Post by tatewise » 28 Aug 2019 21:33

It is never possible to make batch updates with a Query. You can only query the data, and not change it.

You can do that with the Search and Replace Plugin using some of its advanced features.
It may be advisable to create a Backup beforehand.
I suggest you make a few replacements by hand to see how Diagrams and Reports, etc, show that form of Death Date.
Generally even an approximate Date is usually better than no date at all, as it helps when reviewing documents, etc, so consider whether the change is really beneficial.

On the Major Options tab set the Search Scope to Death (INDI.DEAT) so only Death Events considered.
Set the Basic Filters so that only Date fields is ticked, as shown below.
In Search Criteria select LUA Pattern Mode to enable advanced search patterns to recognise years.
Search for ^circa %d%d%d%d$
Replace with "n/k"

FYI: ^ anchors pattern to start of date, %d%d%d%d matches 4 digits of year, and $ anchors it to end of date.

On the Extra Filters tab select No Date Phrase Warnings lower right, otherwise every replacement will produce a warning as "n/k" is a date phrase.

Finally click Search & Replace and when happy that replacements are OK, untick Confirm every item found to allow bulk replacements to proceed. On completion a Result Set lets you check every change and use Edit > Undo Plugin Updates if necessary.

CircaYearToNK.png
CircaYearToNK.png (24.71 KiB) Viewed 978 times
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Re: Batch updates to Date of Death?

Post by Sharon » 28 Aug 2019 23:24

Thank you Tatewise. The reason I want to put n/k in date of death is because it is clear to me that an indivual must have died based on other information/dates I have available, but I don't know the death date, and in a lot of cases don't have a birth date either. I hoping in this way I can then tidy up the Living flag in my database, which currently seems to on/off at random, probably because I didn't mange it properly from the start.

The aim of all this is to get TNG to recognise those who are actually still alive so that their details are not displayed on my TNG website.

I'm using test gedcoms to work through different scenarios to see which produces the result I want. Initially I put in spurious years of death and sometimes also a spurious year of birth in a significant number of records, always using the circa qualifer and making it so that their age at 'death' was 100 so that I could ecognise these records at a later date. It took a long time to do manually and though I can live with made up birth years, I'm finding the made up dates of death misleading and really don't like them in my database. At least putting n/k is a clear marker of what I have done and doesn't pretend more knowledge than I have.

Once I have made these changes I'll run a query to turn off the Living flag of all those with an entry in the date of death nad see if TNG displays them corectly. If not it's back to the drawing board.

I haven't been able to find anything really on the FHUG or TNG websites that gives a recommended way to make these two work together in recognising living people, but I am very open to suggestions if anyone has a better way.

Thank you

User avatar
tatewise
Megastar
Posts: 16469
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Batch updates to Date of Death?

Post by tatewise » 29 Aug 2019 08:57

Do you use the Export Gedcom File Plugin to migrate FH to TNG?
Have you seen the Knowledge Base > Export Gedcom File ~ (TNG) The Next Generation advice, especially the Knowledge Base > Import Details that mention the If no birth date... and If no death date… assumptions.

Also see Knowledge Base > Export to Website Without Sensitive Data that explains how to use the Clean Living Persons Plugin before the Export Gedcom File Plugin. I know it looks a bit complex at first sight because it covers many options, but most users only need a small tailored subset to suit their needs.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
Valkrider
Megastar
Posts: 1064
Joined: 04 Jun 2012 19:03
Family Historian: V6.2
Location: Spain
Contact:

Re: Batch updates to Date of Death?

Post by Valkrider » 29 Aug 2019 09:26

@Sharon

Have you considered what you are going to do with any locations stored against those circa dates? If not, then it may look a bit odd if you have an unknown date but you do still have a place record.

avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Re: Batch updates to Date of Death?

Post by Sharon » 29 Aug 2019 09:51

Yes I used the Export Gedcom file plugin and followed the steps in the Knowledge for TNG and Import details.

I didn't use the Export without Sensitive Data section as that appears to be removing living or private people, and I want there to be at least a box in TNG to show someone exists. Consequently, I didn't use the Flag_Living query.

Having looked at that query more closely, it would have helped me identify the records I have been working on more easily perhaps (those with no dates and whose parents and spouses have no dates either), but it looks as though they would still have had to be handled manually anyway.
The help you gave me for the Search and Replace plugin worked perfectly thank you, so I haven't had to go through the long manual process a second time. I think all my Living flags are now correctly set thank you.

One thing I did notice though, in double checking I had followed the knowledge base correctly - in the Custome Event types list in TNG, 6 Events are marked to ignore on import -Birth, Burial, Death, Divorce, Married and Name. I have no idea why this should be, but the first three may have some impact on why TNG doesn't seem to be recognising Living people correctly, and this may have been exacerbated by the poor quality of my data in respect of the living flag.

Now that I am more confident about the condition of my FH data, I will do some more imports and see what I need to do in TNG to make it work. I may also be able to remove the n/k phrase eventually, using the Search and Replace plugin again.

Thank you for all your help.
Sharon

avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Re: Batch updates to Date of Death?

Post by Sharon » 29 Aug 2019 09:55

@Colin - I don't have any place data for the records concerned, so hopefully that won't be an issue.

Thank you
Sharon

User avatar
tatewise
Megastar
Posts: 16469
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Batch updates to Date of Death?

Post by tatewise » 29 Aug 2019 10:22

The Knowledge Base > Export to Website Without Sensitive Data process and Clean Living Persons Plugin does NOT remove people, but only their sensitive data depending on which options you choose. (If it did remove people the family relationships would become disastrously broken, so it does not do that.)

Please review that process more closely, and it may resolve the whole TNG problem at source.

If TNG is referring to Custom Event Types then I believe that is nothing to do with standard GEDCOM Birth, Burial, Death, Divorce, etc, facts. I suspect you have some Custom Facts in your database that happen to have similar names to the standard facts. So you need to search those out and correct them as necessary. Start with the standard Query for All Facts and sort the Event/Attribute column. That for example will list standard Birth Events as Born... whereas Custom Birth Events will be listed as Birth... and similarly for Died versus Death and Buried versus Burial. Let me know if you need further advice.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Re: Batch updates to Date of Death?

Post by Sharon » 29 Aug 2019 11:28

Thank you Tatewise. I'll have a look at all of those things now. :)

avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Re: Batch updates to Date of Death?

Post by Sharon » 29 Aug 2019 16:03

Hi tatewise,

I have spent most of the day working through the processes for 'Exporting to Website without Sensitive data' - mostly trying different scenarios in terms of Living/Dead flags and having n/k, (or not), in the Death Date field. The results look good, but it has just ocurred to me that a family member who wants to view the tree, and obviously living, will not be able to see the information relating to themselves, which will seem strange.

I have a gedcom ready for importing into TNG, and will certainly try it out, to see how it looks, but I'm not sure it will quite achieve what I want - basically for 'Family' members I know personally to be able to see living data, but to hide this information from 'Guests' who I have maybe come into contact with because we have a shared research interest. Maybe two gedcoms/trees in TNG will be the answer.

I won't have time to do anymore work on this for a day or two now, but with your help I have cleaned up a lot of my data, and have a better idea of how some of the things in FH and TNG work. Thank you.

With regard to the Custom Events, the results from the Fact Set query look as they should - Birth=Born etc. So that looks like I'm using the standard events, which is what I had assumed - I don't think I would have created events that were already there in FH. On the other hand they are in the list of events on the Custom Events page of TNG and marked as Ignore. I'll look into this a bit more in afew days time too.

Thank you again for all your help.
Regards,
Sharon

User avatar
Jane
Site Admin
Posts: 7728
Joined: 01 Nov 2002 15:00
Family Historian: V6.2
Location: Somerset, England
Contact:

Re: Batch updates to Date of Death?

Post by Jane » 29 Aug 2019 16:10

Sharon wrote:
29 Aug 2019 16:03
I have a gedcom ready for importing into TNG, and will certainly try it out, to see how it looks, but I'm not sure it will quite achieve what I want - basically for 'Family' members I know personally to be able to see living data, but to hide this information from 'Guests' who I have maybe come into contact with because we have a shared research interest. Maybe two gedcoms/trees in TNG will be the answer.
On my TNG I load living people, but with no facts. Visitors who are not logged in only see initials and the ones who are logged in see the full names. You could hide living people completely for guests.

N.B The living people I load on are only those who have given me permission to do so, others are not uploaded at all.
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

User avatar
tatewise
Megastar
Posts: 16469
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Batch updates to Date of Death?

Post by tatewise » 29 Aug 2019 17:28

I am not aware of any online family trees, including TNG, where the personal data of some Individuals is visible to 'privileged' visitors, but hidden for 'guest' visitors. So I suspect you will need two uploaded GEDCOM but by using Living and Private flags I think the Clean Living Persons Plugin can produce both versions.

To check for Custom Events open the GEDCOM file that was uploaded to TNG with a plain text editor such as Notepad.
Then use Edit > Find to search for 2 TYPE Birth to find Custom Facts named Birth.
Any that match should be preceded by 1 EVEN on the line before.
Do the same for Burial, Death, Divorce, Married and Name.

Alternatively, search for the lines 1 EVEN and check what follows 2 TYPE on the next line.
It is easier to do that by going to the end of the file and using search Direction Up instead of Down.

If those searches don't discover any 1 EVEN & 2 TYPE lines that match the Custom Event Types listed by TNG then I have no idea what triggers those items.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
Valkrider
Megastar
Posts: 1064
Joined: 04 Jun 2012 19:03
Family Historian: V6.2
Location: Spain
Contact:

Re: Batch updates to Date of Death?

Post by Valkrider » 29 Aug 2019 18:08

@Mike

Genealone, which is similar to TNG, does exactly what Sharon is asking for. It just shows the name of living people but if you have site login credentials and suitable permissions you can see all the living persons data. Unfortunately, it has not been updated for some while and whilst the developers website is still online he has gone awol and does not answer emails. It still works fine though and it is what I still use on my personal website, but I will have to replace it as it is not PHP7.x compliant and the code is obfuscated and so cannot be edited unfortunately.

User avatar
capnkeith
Famous
Posts: 190
Joined: 09 Mar 2009 17:15
Family Historian: V6.2
Location: Suffolk, England

Re: Batch updates to Date of Death?

Post by capnkeith » 29 Aug 2019 18:52

Some clarifications on TNG

It is possible with TNG to protect living individuals and to further protect some to only selected individuals.

Firstly it is possible to limit the site to those who have applied and been given access by you to the site.

Then if you have more than one tree you can apply their access to one or as many trees as you wish.

You can mark all those living so they are not accesible to any that you have given site acces to unless you then give them rights to do so. It is also possible to restrict that down to a particular branch of a tree like the immediate family.

Living indiduals can be flagged within TNG or more easily they can be flagged in FH, Jane has created a query and plugin to simplify this process.

If they dont have an entry in their death field they will be classed as living by TNG, and most other internet sites, but if anything that is put into that field (n/k for example) TNG will class that person as dead. Similarly, and this again applies to several internet sites that host your trees, if you have someone who lived in the 18th century but you have no birth or death dates they will get marked as living. Even if they have several generations of descendents marked as born and died.

Regards
Keith

User avatar
tatewise
Megastar
Posts: 16469
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Batch updates to Date of Death?

Post by tatewise » 29 Aug 2019 19:55

Thank you Keith for that insight into TNG.
Is it easy to reapply those access rights to each uploaded GEDCOM from FH for the same users and family branches?

Your points about Birth/Death Event inclusion or exclusion are well made, which emphasises the importance of at least entering an approximate Birth Date &/or reviewing the If no birth date... and If no death date… assumptions in TNG.
The If no birth date... assumption could be changed from the default of Living to Deceased.
The If no death date… assumption treats a person as Deceased if older than 110 or more or less.
Those then mitigate the missing Birth/Death Event scenarios you mention.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
capnkeith
Famous
Posts: 190
Joined: 09 Mar 2009 17:15
Family Historian: V6.2
Location: Suffolk, England

Re: Batch updates to Date of Death?

Post by capnkeith » 29 Aug 2019 21:44

Hi Tatewise,

As long as you remember to keep your living people flagged. Then all you have to do is upload the tree via the FH Export Gedcom File Plugin. You then use Jane's query in FH to collate all the flagged Living people in the Export File. Run the Plugin which then flags them all in TNG. Next time it's exactly the same all you have to remember to flag them in the first place when you enter them in FH, If it's just an odd one or two you can use the edit function for an individule in TNG. Remembering of coarse to update FH or you will have the same problem next time. This is a l link to the topic where the proceedure is explained more fully.
https://www.fhug.org.uk/forum/viewtopic ... 740#p83909

Regarding Birth Death/Events they are irrelevant if you use the living Flag. If there is no Flag it gets treated as deceased, if there is a Flag it's treated as still living.

If there is any doubt they get a living flag until I can confirm otherwise, or someone sends me a message that if that blank box is for Charlie he has passed on. Which is handy because I then probably get the details as well.

Regards
Keith

User avatar
tatewise
Megastar
Posts: 16469
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Batch updates to Date of Death?

Post by tatewise » 29 Aug 2019 21:58

I wasn't so much asking about setting the Living flag as the allocation of access rights for different classes of user to different classes of data. e.g. Restricting access rights to details in a tree branch to just family members.
When a new GEDCOM is uploaded will those access rights be auto-applied to the equivalent branches of the new TNG tree?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
capnkeith
Famous
Posts: 190
Joined: 09 Mar 2009 17:15
Family Historian: V6.2
Location: Suffolk, England

Re: Batch updates to Date of Death?

Post by capnkeith » 30 Aug 2019 08:33

Sorry, Yes all their settings remain the same throughout updates to the gedcom. Every user has their own page with all their settings on which are either tick boxes or radio buttons. Which include being able to edit the gedcom download parts of it etc.
Keith

avatar
Sharon
Platinum
Posts: 37
Joined: 11 Mar 2004 20:15
Family Historian: V5
Location: Chester
Contact:

Re: Batch updates to Date of Death?

Post by Sharon » 31 Aug 2019 18:13

I've been looking at all the replies since I last logged on, and to the thread highlighted by capnkeith (which I found a little confusing though)

It looks like I'm not the only one with this problem: namely to uplaod a gedcom with all details, so that those with access rights to see Living people can do so, while those with restricted access can only see an initial and surname.

If I have undestood these threads correctly, these are the steps :-
Following the Knowledge base instructions for the Export Gedcom File plugin, and the TNG instructions at https://www.fhug.org.uk/wiki/doku.php?i ... iving_flag

1. Make sure all Living flags in FH are correctly set
2. Download the fh_set_living.php file and upload it to the main TNG folder.
3. Download/Install/Run the living_record_ids_for_TNG query, to return the IDs for all Living persons, and copy the results to the clipboard.
4. Export the gedcom from FH using the Export Gedcom File plugin.
5. Import it into TNG (I did this with the 'Do not reset Living Flag' option checked. I also ran the secondary processes as normal)
6. Run the fh_set_living.php file and enter all the Living IDs from FH into the box. Also enter the name of the gedcom file.

Have I missed anything out, because when I tried to run the PHP file, have entered the IDs, I just keep getting HTTP Error 500.

That said, when I went to check the data, logged in as a guest, then again as Administrator, the data seems to be displaying as I want, using a single gedcom, with the exception of 1 or two deceased records which are hidden as Living, which I can live with for now, though I don't know whether this means the PHP file actually worked, or whether it was because the Living flags were set correctly from the start.

The User Restrictions rights page in TNG allows me to specify the access rights of each individual user, but I still need to find a way to allocate full rights on part of a tree (for example the user's own family) and restricted rights on the rest of the tree. I'm hoping using Branches will help me sort that.

NB - I didn't run the Clean unwanted fields plugin on this occasion as it only ever removed I item in the past, and I didn't run the Clean Living Persons plugin, for obvious reasons.
On the TNG Setup>>Configuartion>>Import Settings page, for 'If no birth date, assume' I set the option to assume deceased.

For now though, I am very happy, 95% of my data is displaying in TNG as I want it, just a little more work to do to specify each users rights to specific part of the tree.

Thank you all for your help.
Sharon (Happy Bunny :D :D )

Post Reply