* creating a file for Ancestry with no facts for living individuals

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

If all your Living people have the Living Flag initially then add that Flag to the Query Columns.
Then when you sort & select, include the Living Flag Column in the sort & select process so only those Name Sources are deleted.

For counting records use File > Statistics.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

I think that is getting a little complicated. I may just stick with my method. I'll have to think about it.

I thought I couldn't open the files bound for Ancestry which are output from your plugin in Family Historian. In any case, after thinking about it, I'm thinking the extra records are the source citations on living people's names so I won't worry about it.

Thanks
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

You could use File > File Statistics on the Gedcom before you run the Export Gedcom File Plugin.
You can run FH on the Export Gedcom File exported Gedcom with some caveats.
Switch off Auto-Save and do NOT use Save, not even when you Close, then the Gedcom should stay intact.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

OK... I'll keep that in mind.

Thanks
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

Bill, I could not resist the temptation to modify the Clean Living People Plugin to simplify your process and check its operation is satisfactory.

So attached is Clean Living Persons Plugin Version 1.2.4 Date 21 Feb 2017 (note the temporary new name).
If any of the Living Names: options are chosen except No Change then this version removes :-
  • All fields subsidiary to INDI.NAME such as Nickname, Name Prefix, Name Suffix, Source, Note, etc.
  • All Alternate Name fields INDI.NAME[2] et seq.
  • All Title Attribute facts INDI.TITL[1] et seq.
  • All <whole record> Sources from both the Individual and Spouse Family records.
To avoid needing your Queries to remove specific fields from non-LIVING records, I have written a Plugin.
See attached Clean Ancestry Fields Plugin Version 0.1 Date 21 Feb 2017 which removes :-
  • All Individual & Family record <whole record> Sources.
  • All fact AGE and Citation PAGE, DATA.TEXT, NOTE2 fields.
  • All Source record _TYPE, TEXT, NOTE2 fields.
[ATTACHMENTS deleted as now in Plugin Store.]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Wow. :o

I've downloaded these and will give them a try. It will be in a couple hours as I have some errands to run this morning.

Thanks so much!
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

I have continued testing, and discovered some other flaws in the Clean Living People/Persons Plugin.
The following rarely used LDS Orinances are not deleted, because they are not strictly Facts:
LDS baptism, LDS child sealing, LDS Confirmation, LDS Endowment, etc.
Also Alias needs deleting as that could disclose a name.
There are a few other data fields, but they probably don't hold sensistive details.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Hi Mike,

As for the new fields you mention, I don't use any of those, but I agree they should probably be added so that if someone does use them, they will be removed. I've been testing out the new versions of the plugins and things look really good. I've only been looking at the files just prior to running them through Export Gedcom File.

My method creates a Gedcom with 378,715 records. Your method (using the new plugins) creates a file with 378,420 records. This is a difference of only 295 records which is really amazing I think. Here are a couple of things I noticed.

My method creates 6,183 families and yours creates 6,180. It looks like my method is actually creating some families that don't exist in the original project. Not sure how it is doing this. I thought you might find this interesting even though it has nothing to do with your method.

The original project has an individual 20196 (? Fure) that looks like this:
image1.jpg
image1.jpg (68.26 KiB) Viewed 11511 times
Here is what my method creates:
image2.jpg
image2.jpg (71.41 KiB) Viewed 11511 times
Your method looks like the original project.

In the original project the parents for 20196 (? Fure) are 18418 (James Arland Fure) and 20194 (? ?). This person, 20194, is James' second wife. There is just one set of parents for 20196 (? Fure).

In the output of my method there are two sets of parents for 20196 (? Fure).

The first set has just a single mother 20194 (? ?). This is the same person as in the original project. Her record ID is unchanged after the merge.

The second set has a couple. The father is 22179 (James Aarland Fure) which is the same as in the original project. The mother is now 22563 (? ?). Who is this? She is the 1st wife of James.

So, the question is why aren't 22179 and 20194 his only set of parents. Why is the 1st wife, 22563 (? ?), ending up as the mother of 20196 (? Fure)? Why is the 2nd wife, 20194 (? ?), losing her husband in one set?

EDIT: I ran this again and the same problem did not occur. Very strange. Is it an intermittent problem with the merge? I may look at it a bit more, but I think I'll switch and use your new method Mike. No extra set of parents in my testing!

I'm not sure what all of the 295 record differences are, but I'm not worried about it at this stage. I'll keep looking at it and see if I can identify what the other differences are. For my purposes, your version is working great for me.

Are you going to be publishing the Clean Living Persons plugin for others to use as well or will you work with Jane to update her plugin?

I really wasn't expecting you to create/modify these plugins for me. I really appreciate it. Thanks so much!
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

Something in the Merge is not reliably matching the Individual &/or Family records for that example.
Cannot explain why it should be inconsistent.

Ideally Jane should adopt my Clean Living Persons plugin and integrate its ideas into her Clean Living People plugin.
She may want to add options for conditional removal of groups of the extra fields we have included, and incorporate David's requests in Clean Living People plugin question (14697).

In the meantime I'll make both my Plugins available within how_to:exporting_gedcom_with_multimedia#export_to_website_without_sensitive_data|> Export to Website Without Sensitive Data.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
davidm_uk
Megastar
Posts: 740
Joined: 20 Mar 2004 12:33
Family Historian: V7
Location: St Albans, Hertfordshire, UK

Re: creating a file for Ancestry with no facts for living individuals

Post by davidm_uk »

Not sure if it's best here or in a separate post, but for any new version of Jane's Clean Living People plugin can I add my request for an option(s) to be able to retain individual primary full name, and year of birth and marriage, but remove everything else for living people, (including source records cited for living people even if they are also cited for non living people).
David Miller - researching Miller, Hare, Walker, Bright (mostly Herts, Beds, Dorset and London)
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

David, that would be better in Clean Living People plugin question (14697), which I notice you already did on Saturday.
However, you need to be as explicit as possible about what you require, and leave nothing undefined.
You say nothing about subsidiary NAME fields such as NICKname, etc, or Alternate Name fields, or Aliases or Titles.
When you say remove everything else does that include Sex, Custom Id, and many other non-Facts?
It could even be taken to mean Parents and Children.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

Thanks. Very strange indeed. If I can get it to consistently have a problem, I'll report it to Calico PIe.

Here is my write-up on using the new method with the new/modified plugins. Maybe someone will find it useful.

Thanks
Attachments
Create file for uploading to Ancestry - no-merge method.docx
(22.76 KiB) Downloaded 166 times
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

Bill, can you review that document, because I think it has some issues.

Most significant is that Step 2 Split Tree Helper removes all Flags that may be crucial for the Step 4 Clean Living People/Persons Plugin that tests Private and Living Flags.

Secondly, your original create & merge method only removed Note and Repository records associated with Living people, whereas now they are all removed unconditionally.

In Step 5 Export Gedcom File the Other Options tab settings are superfluous as all AGE fields and Submitter records were removed earlier.
The TITL Ω File Root deletion should be superfluous if File Root 1 _ROOT: Remove entirely is chosen in Extra Options tab.


Could you try using the method in how_to:exporting_gedcom_with_multimedia#export_to_website_without_sensitive_data|> Export to Website Without Sensitive Data which uses a different order for the steps.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Hi Mike,

I wondered about whether or not the Living flags were necessary for Clean Living People/Persons. Since the plugin requests a value for set if no death date and born after:, I assumed it was making its own determination of whether a person was alive or not and that it didn't use any existing living flags on the file. In any case my living people ended up being processed properly by the plugin... all were given the name 'Living' and things were cleaned as expected.

I thought I was always removing the note and repository records. In my merge method I deleted them in the Split Tree Helper in Step 1 and Step 2. In my final output I don't have any repository records. Actually at this point I have no note records in my FH file anyway so I can't tell if they are all being removed or not.

For the Others tab options in Export Gedcom file, I admit these are redundant now. I had the age ones in there earlier because I hadn't done the query steps to get rid of them yet. Now that I have your plugin, I really don't need these changes to the options. I don't know why I had ever changed the Header Submitter 1 SUBM option as the records are being removed as you indicate. I guess I missed the File Root 1 _ROOT option. I will update my write-up for these changes.

As for running them in a different order, it looks like the only difference in your write-up in the KB is that the Split Tree Helper step is done after Clean Living Persons rather than before Clean Ancestry Fields as I am doing it. Should that make any difference?

In any case I will run your no-merge method and my no-merge method and compare the two output files.

Thanks
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

OK... here goes. I ran the two tests and these are my findings. I was looking at the file just before it went in to Export Gedcom File. One difference that I made to your method was to remove ALL note, media, and repository records in the Split Tree Helper step rather than just those with no links because I don't want any of these in my file. I did run the steps in the order you did... 1. Export, 2. Clean Ancestry Fields, 3. Clean Living Persons, 4. Split Tree Helper, 5. Export Gedcom File.

1. Your method ended up in more people being flagged as 'Living' than my method did. Of course because you marked them as 'Living' their citations and facts were deleted which resulted in fewer records in the Individual section of the GEDCOM file. Some of these people were individuals that had no birth or death dates but were married to folks born after 1910 which is the cutoff date I use. I do NOT tick the option in Clean Living Persons to Assume living where no life dates.

2. Your method ended up in more records being deleted from the Family section of the GEDCOM because some of the extra 'Living' individuals you flagged in #1 were husbands or wives in the families so the facts and citations were deleted from the marriage.

3. Your method did not get rid of sources with 0 citations so that all sources were still in the file. I'm not sure why this happened. I ran out of time tonight so I will try again tomorrow to see if I get the same results. I thought I selected the option to exclude sources left with no links to them in Split Tree Helper, but maybe I missed that one.

What is amazing is that the files had about 380,000 records total and there was only a total difference for all of the above reasons of 297 records!

There are a couple of things I don't understand which may be bugs.

4. Neither your method nor my method removed any '2 NICK' records from the file. Shouldn't this be done in Clean Living Persons?

5. Both your method and my method ended up having '0 @P...' records in them. Shouldn't these have been removed in the File > Import/Export > Export > GEDCOM File... step? I checked the option to exclude Place Records. In any case, after running the file through Export Gedcom File and using the 'Place Record 0 @p%d+@' option of 'Remove Entirely', they were stripped out.

Am I doing something wrong in regards to #4 and #5?

Thanks
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

It is all looking pretty good, and different users will prefer different settings.
You are correct, you did delete all Note & Repository & Media records unconditionally (my memory failure :oops:).

1. & 2. Clean Living People/Persons has (Private Flag Set) and (Living Flag Set) options stated at the top.
If Split Tree Helper deletes all Flags beforehand, then neither of those Flag Set options can apply.
It is only if Force Living Flag is ticked that the three options below come into play. That is what you DOCX method relied on.

3. Both Clean Living People/Persons and Split Tree Helper should delete Source records with no links, so cannot explain why any 0 Citation entries remain.

4. Yes, 2 NICK should be removed by Clean Living Persons , but only for LIVING people. The Name fields of all others are unaffected. This is different from the Queries method that removed subsidiary Name fields unconditionally.

5. The Place records would have been deleted by Export > GEDCOM File, but as soon as you open that Gedcom in FH it automatically creates a blank Place record for each Place name. So the subsequent Plugin & Split Tree Helper steps will Save the Gedcom with not only those changes but also the empty Place records. That is one of the reasons why it is risky to run FH on the Export Gedcom File Plugin's exported Gedcom.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

1 & 2. I think I understand, but I just need to make sure. So, if I have the Living flag set the way I want in my GEDCOM and I do NOT check Force Living Flag then the plugin will use the values already set for the flag? Is that right? So, there is really no need to use the Force Living Flag option if I already have correct values in my GEDCOM. I still don't really understand why your method ended up with more Living individuals, but I can live with it.

3. Very strange. Using your method the citations with no links are definitely there. I can see how I might have missed the option in Split Tree Helper, but there are no options on Clean Living Persons. I'll run some tests later today and look at the GEDCOM after every step to see if I can figure out where things are going wrong.

4. I hadn't realized the distinction. This is actually good. I can see they only remain for non-living individuals and these could be good to have. This is an improvement over the query method.

5. I think I understand, although I'm not sure what you mean by "blank Place record". An example looks like this:

0 @P2887@ _PLAC Bremerton, Kitsap County, Washington, USA

I will put a note in my write-up to explain this as it is a little confusing to remove them in one step and see them still there in later steps.

Thanks
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

1. & 2. Yes, IF you have the Living flag set the way you want in the GEDCOM and and don't tick Force Living Flag then the Clean Living Persons Plugin will simply honour the Living flags. BUT that is a big IF, because if you forget to set a flag then that person will not get cleaned. Also David has asked for, and I am working on, cleaning Spouses and Children of Living persons, regardless of their status, i.e. that would include deceased Spouses and Children.

3. In the Clean Living Persons/People Plugin there is a Remove unused Source Records option at the bottom, which executes after all the other cleaning has completed. It would be useful to know at what stage the 0 Citation cases get overlooked. My tests suggest the Plugin does delete all 0 Citation Source records.

4. OK, good.

5. A Place record can hold all manner of 'stuff' such as Standardized, Lat./Long., Notes, Media, etc.
That is all deleted when Place records are deleted, and may include Place records not linked to anything.
The blank Place record simple holds the Place name from the Place fields, so discloses nothing new.
This is the default FH action when opening a Gedcom, because Place records are essential within FH, but are a non-standard extension, and most other products do NOT have them, which is why Export Gedcom File changes/removes them.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

1 & 2. I do keep my flags up to date and will have an initial step in my write-up to make sure they are set properly anyway. If you add those functions for David, I hope you add them options. I wouldn't want to do that myself.

3. I do always have that option ticked. I am still trying to figure out what is going on and will let you know once I've figured it out.

5. OK. I see what is going on. Thanks for the explanation.

Thanks!
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

1. & 2. That is a brave statement. I could provide a temporary version of the Plugin that listed any Individual records found by the Force Living Flag option and therefore don't have the Living flag set. Remember the Plugin also caters for users who do not maintain Living flags. Yes, David's variants will have options.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

1. & 2. I agree it is a bit of a brave statement. But I will always have my write-up in front of me when I do this process and it will have that step in it so I think I'll be OK. Just for grins you could provide that temporary version of the plugin and I can see what difference it would make on my file.

3. OK, I figured out where the un-linked sources were coming from. Here are the total number of sources and the number of un-linked sources at the different steps of the process:

original file, 2933, 86
after export, 2847, 0
after Clean Ancestry Fields, 2847, 1
after Clean Living Persons, 2810, 0
after Split Tree Helper, 2810, 85

Don't let the fact that 86 - 1 = 5. This is strictly coincidental (although it threw me for awhile). The 85 sources with no links at the end are totally different than the 86 at the beginning.

Clean Ancestry Fields was turning one linked source into an un-linked source. The source had 6 citations on 6 different individuals. All 6 links were whole record citations. These links were removed as expected. This one source just happened to have all of its links removed in this fashion. This un-linked source was then getting removed by Clean Living Persons.

Split Tree Helper was turning 85 linked sources into un-linked sources. I looked at about 15 of these and all of them had only one link prior to running Split Tree Helper and the link was to a note. So, when Split Tree Helper deleted all notes fields within records, it left these sources there un-linked. So I guess Split Tree Helper is doing the Delete all Source Records left with no links to them processing before it does the Delete all note fields within records processing. My method was not creating these un-linked sources, because it had Split Tree Helper as the second step instead of the next to last step. These new un-linked sources are not related to living individuals, so I'm not terribly worried about them.

Phew... that only took me about 3 hours to figure out. :D

Thanks
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

Well done!
That sounds like a bug report to Calico Pie regarding Split Tree Helper is called for.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

I will let them know.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: creating a file for Ancestry with no facts for living individuals

Post by tatewise »

Here is that temporary Plugin, with the extra options I have been working on, but you don't need to use them.
(Anyway the if spouse or parent is alive option is not working yet.)

You might want to Rename your current Plugin so this Attachment won't destroy it.
Then afterwards, delete this temporary version, and Rename your current one.

Alternatively, you can let this Attachment overwrite the current Plugin, and either just keep it, or revert to the download in how_to:exporting_gedcom_with_multimedia|> Exporting a Family Tree with/without Media.

When you run this Plugin you do not even have to OK the last step, it will still give a Result Set of Individuals who are deemed Living but without the Living Flag.

[ATTACHMENT deleted as now in Plugin Store.]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: creating a file for Ancestry with no facts for living individuals

Post by BillH »

Mike,

At first glance, it seem to be forcing a lot of people as Living that I don't consider living and are not flagged as living by my method.

1: ? /Gauthier/ has no information other than just a name. He has no children or parents. He has a wife who is dead (1904 - 1987). Why is he being forced to living?

2: Patricia H Henshaw has no other information. She has no husband and no children. Her father is dead (1892-1980) and her mother is dead (1896-1969). Why is she being forced to living?

There are a lot like this.

Also I have situations were you are not forcing to living, but my method does have them as living.

3: Oliver Strom has no information other than his marriage (1951). He has no parents, but he is married. His wife is dead (1921-2013). He has a child, Martin, but I have no dates for him. I would think Oliver might be considered living, but he is not being forced by your plugin. Neither is his child. Both end up being living using my method.

Then there are some I really can't figure out.

4: Virgil William Niewedde was born 14 Feb 1916 and died "bef 12 Sep 1966" yet he is being forced to Living. Why??? He is dead!

That is just a quick glance. There may be other scenarios as well. Your temp plugin is throwing up about 650 folks that would be forced that my method doesn't make living.

I do like the option of forcing folks to living if their spouses are living as I am not catching some of these using my method. So I'm a bit confused now over which method to use.

Here are the options I used in the temp plugin. Just a thought... any chance of getting a help button with in-depth help on what each option means?
image1.jpg
image1.jpg (119.66 KiB) Viewed 11423 times
Thanks
Post Reply