* Updating existing FH records with data from CSV

[NickiP]

I know there is a plugin for creating new people from a CSV file (Flexible CSV Importer) but there doesn't appear to be anything to update existing FH Records with data from a CSV file, or not that I can find from searching.

I would like to export my tree to Ancestry but will have issues with indexing on the site due to the fact that, unlike most FH software, they won't use baptism/burial dates as a substitute for birth/death dates in their indexes which will leave my 60k+ tree being unindexed and therefore of no use for DNA match checking. Where I don't have a confirmed birth and/or death date but have a baptism and/or burial date, I use the latter and leave the former blank because it looks better in narrative reports which is my main use of the software for presenting my research. My intention would be to update the data on a copy of the project once in a while and upload to Ancestry.

I can run a query to pick up all individuals with no birth date but who have a baptism date (and the same for death/burial), and include RECORD ID, and then export to a CSV file and use Excel to manipulate the data so that the baptism/burial year could be added to the birth/date cells.

I'm just wondering if there is then any way of then adding the birth/date dates to the existing records (using Record ID) in Family Historian from the CSV file?

I know it's possible to do something similar using MS PowerShell for Active Directory and Skype for Business as have done it at work. However, my coding skills are limited (as are my Excel skills ) so I'm going to struggle to even know where to start with plugins.

I'm just wondering whether anyone has anything that could be adapted to achieve this please? Lets face it that's what happens with much using PowerShell.

Thanks

Nicola

[tatewise]

The Flexible CSV Converter does sync with existing records via the Custom Id.
So use Tools > Record Identifiers to copy Record Id to Custom Id and use those Custom Id in the CSV and that should work.

[NickiP]

Thanks Mike, I obviously didn't read the notes correctly. I'll take a look and give it a go.

[tatewise]

Since this is an Export to Ancestry strategy, I wonder if the Export Gedcom File plugin can be adapted to help you.

When exporting to (FTD) Family Tree DNA the plugin synthesises missing Birth and Death events from Baptism, Christening, Burial & Cremation events as explained in the Help page https://pluginstore.family-historian.co ... y-tree-dna.
That is for the same reason that Family Tree DNA does not recognise anything but Birth and Death events.

When exporting to (ANC) Ancestry Family Tree the plugin could be adapted to similarly create synthetic Birth & Death events.
Maybe it needs to become a conditional plugin option for exporting to any product where Birth and Death events are essential for DNA features to work.

[NickiP]

Hi Mike

Apologies I've only just seen your reply. It's taken me this long to get a single project file ready to use due to all the merging of places etc that I've had to do. Suffice to say, I'm going to keep everything in a single project now even though it's now taking 30 seconds to load or reload the file in Ancestral Sources in spite of having an SSD. I only use AS for adding Census entries so will have to put up with it. Perhaps a good excuse to get a new laptop with an M.2 SSD, my laptop is admittedly going on 9 years old in spite of various upgrades.

If it would be possible to adapt the Export to Gedcom file plugin, that would be a lot quicker for me. It would also benefit anyone who just wants to upload a copy to Ancestry anyway not just for DNA and has the same issue with missing birth/death years.

Does the export to FTDNA leave the baptism/christening and burial/cremation events in place or remove them after creating the new Birth/Death events please? I can't quite work that out from the details so my apologies if I'm being stupid. Obviously I would like them to remain as am going to make the copy tree available to close family, albeit I'm not planning on uploading with all events that I have added to Family Historian. Also does the FTDNA option also copy the place details or just the date? Personally I'd only want to make use of the date and no place/location.

In addition to burials and cremations, would it be possible to also include Probate dates where no burial or cremation date exists please as I've a number of individuals with only Probate dates due to missing burials?

Thanks for the interest.

Nicola

[Mark1834]

Nicola - the disadvantage of the Flexible CSV Converter and Export GEDCOM plugins is that they are relatively complex. That is inevitable, as they are designed to cope with lots of different scenarios.

I take a different approach for my export to Ancestry. I have a dedicated plugin that does all the processing in one step, without having to remember multiple menu options.

• Extracts just key events, such as birth, death, baptism, burial, census, residence, probate, marriage, etc, with just "who", "when", and "where".
• Removes completely any reference to living people or any other individual marked as private, rather than just hiding names and dates.
• Removes all the extra detail that I don't want to share with Ancestry, such as sources, media, detailed notes, etc.

For DNA matching or "cousin bait", that's all you need.

The plugin was designed originally for syncing with Ancestry via RM in order to exploit Ancestry hints, but its GEDCOM file can be uploaded directly to Ancestry (or any other similar site) if required. There is a description of the full sync process and a link to the plugin in this Knowledge Base article, but only the first couple of sections would be relevant for you - ignore all the RM details. It doesn't create synthetic birth and death events from baptisms or burials/probate, as I always include approximate values in my database, but it would be simple to add that as an option if required.

Is that of any interest?

[ColeValleyGirl]

Mark, thanks for mentioning the need to remember various options for the Export Gedcom plugins for various destinations.

Mike, I may have missed this functionality if it already exists (and if so, apologies) but it would be really useful to remember the customised options I've used for different destinations and 'reload' them when required.

[tatewise]

Nicola, currently the FTDNA conversion replaces Baptism, Christening, Burial & Cremation events with Birth or Death events, reduces the Date to a year, and I think it retains the Place & Address, etc.
As Mark says, trying to cater for every variant on that theme that anybody might want and different products might need does get complex, and needs a potentially confusing set of options for users to select.
If a single set of rules could be agreed then only one enable/disable option is required.

Helen, the Export Gedcom File plugin does remember the separate settings chosen for each GEDCOM Export Mode: target product on a per Project basis. That is generally the case for all my 'complex' plugins. However, if you create a new Project, or work on an exported standalone GEDCOM file, then that does not help. On reflection, it might be better to save the settings globally so they get applied to all Projects and standalone GEDCOM in the same way. What do you think?

[ColeValleyGirl]

Mike,

Saving settings globally would be a great help -- starting from scratch in a new project is a pain at present. However, the ability to have 'named sets' of customised settings would also be good -- for example, if you're experimenting with settings for one of the export modes, but want to remember 'a set that works' that you can revert to if things go wrong... Or you want different settings for different instances of the same destination.

[Mark1834]

My core application is RM sync, which relies on a consistent export. Therefore, I will keep my plugin “one click”, with the export file name as the only option.

However, I think there is a place for a simple GEDCOM export routine that takes core data (the “who, what and where” of key events described above), with just basic configuration options. Leaving out all the complex stuff that varies so much between apps makes it much more generic in its application. There must be half a dozen different plugins that do various bits of this, so it can be confusing for users to know which one to use and how to set it up.

If your needs are simple, use the simple plugin. If you want fine control of the detail, use the complex ones.

Any merit in pursuing that?

[NickiP]

Mike, I can appreciate how difficult if not impossible it is to try and include all variants that people may like in a plugin. Unfortunately the FTDNA conversion won't work for me because I want to retain the existing events and just add a new event for the others from that data. I always like to complicate things.

Mark, if you are happy to adapt your existing one click plugin to add synthetic birth/death dates then yes please I'd be very much interested. I've used the tree splitter to remove all custom events and others I don't want to upload, along with media, sources etc so for me it's a matter of getting a birth/death date created from the other events and then export to gedcom.

I know probate dates aren't necessarily an indication of a death date as some can take place years after the date but it's better than nothing where both the death and burial date are missing.

I'm slightly in two minds as to whether I'd want a place included with birth/death dates because of the way I use places in FH. I don't use the address field so all details of a location are in the place field, such as full church name or obviously Probate court. In the case of using probate to create a death event, it'll look slightly stupid saying someone died in a probate court. I just decided to use this when I started with FH because I found it easier when customising the sentence templates for narrative reports. As I said previously, I do like to complicate things. I know with Ancestry and their hints a place is required (with country) to try and get a more accurate hint (accurate as Ancestry's hints are (not) due to the way they are created) but as this isn't the reason for the upload, I suspect it may not be as critical for DNA results. Albeit I may find I'm wrong with that.

Thanks

Nicola

[Mark1834]

There may be a misunderstanding here. I’m not familiar with the FTDNA plugin, but it would be perfectly feasible for synthesised events to only be in the GEDCOM export file, and leave your database unchanged. I agree that the help may need rewording slightly to clarify that point, but Mike will be along later to confirm exactly what the plugin does.

[NickiP]

Sorry Mark, I think there has been a misunderstanding then.

I thought you were talking about adapting your existing plugin rather than Mike's:

It doesn't create synthetic birth and death events from baptisms or burials/probate, as I always include approximate values in my database, but it would be simple to add that as an option if required.

Is that of any interest?

I only want the synthetic birth/dates in an exported gedcom, not in any Family Historian file. The FTDNA conversion option on Mike's plugin won't be of any use to me as it converts to birth/death events and removes the existing baptism/burial events which I want to keep.

Edit - Just to confirm what I'm trying to do is add a birth/death date from either a baptism/burial or probate event so that Ancestry has something to use to index the tree as they don't use baptism or burial events in lieu of birth/date events. I have no interest in adding them to my existing FH project. However, as I don't have burial dates for all individuals where there is no death date, I'd like to use the probate date in lieu of a burial date in those circumstances where one exists. I also only want to use the year and not the full date and not the place/address.

Alternatively, it would be just as useful to me if there was a plugin that would created a synthetic birth/date death from these events in an existing FH project, then I could just run the normal export plugin to create the gedcom. I'm using a copy project anyway to remove everything I don't want to upload.

[Mark1834]

Clear, thanks. It’s a fairly simple modification to my plugin to add synthetic birth and death, so I’ll do that and post it here later (probably tomorrow morning). It will be a “one off” special customisation, but it might serve as a general prototype for the type of simple exporter I described earlier. Core events only, and the option to include or exclude places.

[NickiP]

Thanks Mark, much appreciated.

[Mark1834]

Modified plugin attached. It is basically my RM-Ancestry sync plugin with the RM bits taken out and the ability to synthesise missing births and deaths added. It has the bare minimum of options, just the output file name and whether to include places or not.

The list of events that it exports is fixed and based solely on my preferences for Ancestry . Namely BIRT, BAPM, OCCU, RESI, CENS, RETI, EMIG, IMMI, NATU, DEAT, BURI, CREM, PROB. Census is converted to Residence without asking.

It does not write anything back to your project, but as ever try it first on a copy just in case I messed up! You could run it on one of your extracts, or on the original project. Does it do all the cleansing you would need?

If this does go on to become a general simple exporter, I would add a basic menu to give a minimum of options (event selection, census conversion, synthesise missing births and deaths, etc).

A couple of caveats - I have modified how it saves the output to make it more flexible for non-English characters, but that does mean it is FH7 only, unlike my original. Also, it should be seen very much as a prototype created from another plugin - there are certainly areas where the code could be tidied up for a final version...

(Attachment deleted - updated version in later post).

[NickiP]

Very many thanks Mark.

[NickiP]

Hi Mark

Firstly many thanks for doing this, it is very much appreciated.

It works great for creating births from baptisms as far as I can see, not come across any issues.

However, there is an issue with creating deaths from either burial or probate dates. It is creating death dates even where no burial or probate date exists and the dates for these particular entries seem random a lot of the time.

example 1.JPG (99.97 KiB) Viewed 5533 times

William Rushbrooke and Anna Jardane both have burial dates for the years created for their deaths. As does their son William. However, there are no burial or probate dates for any of their other children but the plugin has created them.

example 2.JPG (130.67 KiB) Viewed 5533 times

Similarly while John Goodman has a probate for 1705 and his wife and daughter have approximate death dates in the original, it's created a fictitious death date for John Bird and all their children apart from the youngest Joshua who was buried in 1739.

It's not just where there are a lot of missing burial or probate dates. I have one unnamed child born in 1839 who has been given a fictitious death date of 1914. There is no known death or burial date for him so it's blank in the original. His parents and siblings all have a death date or one created from a burial so why it's created this I don't know. It's strange.

I did try editing the plugin in notepad to use either burial or probate dates (separate edited plugin for each) but the same issue arises so it's not due to using either.

Nicola

[Mark1834]

Oops, sorry about that! It's probably a quick fix, so I'll proofread the plugin over lunch and post either a corrected version or instructions on how to fix later today.

[tatewise]

Helen, regarding your comments yesterday (I was out all day) making the settings global seems a reasonable change.
It won't affect users with only one Project, and those with multiple Projects are likely to want the same settings.

To experiment with custom settings use GEDCOM Export Mode (CEA) Custom Export Alpha or (CEB) Custom Export Bravo.
These default to (Std) Standard GEDCOM Release 5.5 and (Str) Standard GEDCOM Release 5.5.1 modes respectively, and allow two other export modes to be customised.

I guess it would be possible to add a settings management tab to the plugin offering the following features:

• Allow settings to be local to the Project or global, and manage copying between the Project settings and global settings.
• Allow settings to be copied from one GEDCOM Export Mode to another, e.g. from (ANC) Ancestry Family Tree to (CEA) Custom Export Alpha and vice versa.

[Mark1834]

Sorted - the code for death is slightly different to that for birth as there are two different scenarios (burial or probate), and I had omitted to reset the fake value for each individual. As a result, values leaked between individuals. A new line 157 to declare the variable within the loop and modification where it is used keeps everything fully isolated. I had tested it with too simple a dataset.

It's worth posting a full corrected version for the benefit of anybody else testing it who might not be comfortable making their own corrections.

[ColeValleyGirl]

Mike, copying from one export mode to another would make it easier to experiment.

[NickiP]

Thanks Mark, that works a treat.

Just doing yet more data cleansing on my main project and will then export and upload.

Thanks for your help, very much appreciated.

Nicola

[Mark1834]

Thanks for testing. I think there is a place for a simple extract plugin like this, but I won’t do a smartened up version yet. I’ll be updating the Ancestry sync plugin later anyway, so it will make sense to do them together (or even combine into one).

[tatewise]

IMO:
The Birth should be derived from Baptism and if none then Christening and perhaps even Christening (Adult).
The Death should be derived from Burial and if none then Cremation and lastly Probate.