Page 1 of 1

TNG sync

Posted: 03 May 2015 15:10
by rtilyard
My number one wish is the ability to automatically sync my FH data to my TNG website.

Re: TNG sync

Posted: 03 May 2015 16:04
by tatewise
Do you know there is the Export Gedcom File Plugin to export your FH data & media in various TNG compatible formats that are easily uploaded?

Several FHUG members use it regularly.

See Export from FH6 and Import to TNG (12513).

I think it unlikely that automatic synchronisation with any other genealogy product is going to happen in the foreseeable future.

Re: TNG sync

Posted: 03 May 2015 16:23
by rtilyard
Thanks, yes, fully aware, but who needs to manually keep uploading gedcom files and applying them when we have computers that can do that repetitious labour for us.

I don't do manual backups, my data is automatically synced to dropbox and google drive.

I have my own online store and an eBay store. Sales and listings are automatically synced between the two.

There's no reason for us not to apply the same technology.

Kind regards, Rod

Re: TNG sync

Posted: 03 May 2015 16:38
by rtilyard
Mike, you said "I think it unlikely that automatic synchronisation with any other genealogy product is going to happen in the foreseeable future."

Sorry but I have to respectfully disagree. We need to drag our hobby out of the dark ages where we are stymied still by the failings of gedcom. It's actually far simpler programming in this case to ignore gedcom and directly update the online database. No mistakes in translation and changes are applied immediately instead of weeks or months later.



Re: TNG sync

Posted: 03 May 2015 16:57
by tatewise
I respectfully have to question your over-simplification.
What you suggest is only possible if the other product (TNG in this case) publish an API (Application Program Interface), and most do not, usually because they want to retain intellectual and marketing rights.

I am aware of some the difficulties here because my Plugin Lookup Missing Census Facts tries to provide Internet URL searches for Ancestry and FindMyPast, but they do not publish their interface schema and often change it without notice, as it is only meant to be used manually via a browser.

It does not take weeks to update TNG via Gedcom; just a few minutes.

Even with the Gedcom standard, genealogy products do not implement it fully.
The problem with integrating data via synchronisation is exactly the same.
c.f. The problems with importing TMG Witnesses into FH, and of exporting Face Frames of Images from FH, which certainly impacts TNG.
If two products do not support compatible data schema they cannot be easily synchronised without loss of data.

Re: TNG sync

Posted: 04 May 2015 00:53
by rtilyard
Hi Mike,

Why use a sledge hammer to crack a walnut?

I said all that was needed was to "directly update the online database". Here's the easy way:
  • FH writes all changes to a log file.
  • Log file is automatically copied to your 'cloud' account (dropbox, google drive, etc).
  • FH writes a script you place on your server and run as a cron job.
  • It periodically accesses the log and updates the TNG database.
The TNG database schema is publicly available, no IP to consider.

I agree that it only takes a few minutes OF COMPUTER TIME to update TNG via gedcom. But it may be weeks between the time you add data to FH and when you go through the long-winded process of:
  • producing a TNG compliant gedcom
  • copying it to your TNG server
  • setting your website into maintenance mode
  • running the gedcom update (this is the little bit that only takes a few minutes)
  • checking that everything worked as intended
  • make website available again
  • forward a few weeks...repeat...repeat...
Cheers, Rod

Re: TNG sync

Posted: 04 May 2015 11:16
by tatewise
That is an interesting approach, but the big challenge is the mapping between schema.
The GEDCOM route at least provides some common ground.

To understand what I mean take a look at Knowledge Base > FH GEDCOM Extension List which is effectively the FH schema because it uses GEDCOM as its database.

Also look at Knowledge Base > Export Gedcom File ~ Output Formats that explains the mapping from FH to Standard GEDCOM and the additional mapping for TNG.

The most challenging parts include mapping Witnesses and Place records, choosing character encoding, handling Media Files especially Face/Detail Frames. So creating a log file is only a starting point. Where would you see the mapping being performed? Would it be performed before creating log file so the log is already TNG compatible, or in the cron job? How would updating Media files be handled, because without those the Media record links won't work? Presumably there would be user options to choose various options such as the character encoding, or would the cron job translate from Unicode to ISO or UTF-8 depending on TNG settings?

A variation of that approach might be to run the Export Gedcom File Plugin (perhaps on an automatic schedule) and post the output files instead of a log file. Then a I suspect a cron job could import the Gedcom, and I think there is an option to only process changed records, but I am not a TNG user.

Re: TNG sync

Posted: 04 May 2015 13:45
by rtilyard
Hi Mike,


I don't think I want to propose the entire solution for FH :) who from observation are a rather talented bunch and can easily achieve this.

I think FH must already utilise a log of sorts for multiple undos. This may be a memory array right now, but could be easily exploited for a sync to TNG feature. That implies the log is in FH native format and all the transitioning work would be managed by the PHP script back on the TNG server. It also keeps FH slick and quick, let the server do all the data and CPU crunching.

I would hope that it could be completely seamless without the user having to make choices. If there's a need for any options to be set, these would be best managed in the FH app and could be written to the log file as a header record.

As we're using 'the cloud', all media will be automatically added or updated to the user's dropbox/onedrive/google drive account along with the change log. The server script would retrieve the log, make a first pass through it and copy any required media files from the 'cloud' account to the user's specified TNG file paths, then a second pass would update the TNG database. Processed log records would be time-stamped.

Understanding the TNG schema I think is a no brainer. Just a short few hours analysing the tables and experimenting with some PHP mysqli queries on a test database should be all that's needed. Having said that, it would be in FH's interest to approach Darrin Lythgoe anyway and try and get some collaboration going. You never know, he may be even willing to help with coding the server side. There's a lot of kudos and brownie points in this little project for both parties.

In closing, you said "The GEDCOM route at least provides some common ground."

Tongue in cheek, that's about 10% common ground :P. Every software provider makes their own interpretation of the GEDCOM quasi standard. It simply stifles innovation and wastes countless man years of programming. It becomes more and more irrelevant with each passing year. Heck, on the one hand, as a gesture to be seen as GEDCOM 'compliant', FH does not provide styles for notes, even though EVERY other piece of family history software on the planet has a minimum of bold, italic and underlining. Now in FH6, through market pressures, we see an abundance of 'non standard' tags provided. Amen.

Cheers, Rod

Re: TNG sync

Posted: 04 May 2015 15:46
by tatewise
Hi Rod,

That certainly will give Calico Pie something to think about in an interesting new direction.

I think you underestimate the influence that Gedcom has had on genealogy products.
Although I agree most products abuse the standard, it nevertheless provides some cohesive design concepts that appear in most products. The concept of Individual, Family, Note, Source, Repository & Media records seem ubiquitous, along with a fairly consistent set of Facts and associated Date, Age, Place & Note details, plus of course Citations of Sources. Without that level of conformity any schema mapping would be horrendously complex! It is only really the devil in the detail that we have been discussing. So I maintain that without Gedcom we would hardly be able to start the discussion.

Re: TNG sync

Posted: 06 May 2015 21:56
by AdrianBruce
tatewise wrote:...So I maintain that without Gedcom we would hardly be able to start the discussion.
Totally agree Mike. (Sorry - probably pointless contribution but here goes...) Suppose there are 30 genealogy programs. Without GEDCOM there would need to be 30 x 30 interface routines written across the industry. 900. 30 of which would be in each program.

With GEDCOM in the ideal case (laughable concept), each genealogy software author writes just 2 - export to GEDCOM and import from GEDCOM.

With GEDCOM at least the basis for the data model is roughly consistent, even if the detail can be flawed. Imagine trying to interface Family Historian to a genealogy program that didn't have the concept of a family but had roles in birth events instead. GEDCOM has reduced the likelihood of that.

Syncing to TNG with code written for TNG is a point solution that is useless for any other purpose - e.g. useless for syncing to FamilySearch FamilyTree. Syncing to TNG via GEDCOM might have use elsewhere - were it possible!

But I do also have to say that an alliance between FH and TNG would be attractive, especially if the online stuff were made simple enough for the people who don't understand the concept of a server, and a dedicated interface for that purpose could be worth it.

Re: TNG sync

Posted: 18 Apr 2016 20:59
by cjdenbow
This is a feature that would interest me greatly, if ever developed. I suspect that Darrin, the developer of TNG, would be open to this kind of an alliance, as I think it would make his product more attractive to more people and would increase his sales. Seems to me it would be a win-win situation with both FH and TNG benefiting from increased sales and increased revenue. Why don't you guys check it out? Couldn't hurt! -- Carl

Re: TNG sync

Posted: 18 Apr 2016 21:21
by tatewise
Carl, I think you misunderstand who we 'guys' are. This is the Family Historian User Group (FHUG), that is run by volunteers completely independently from the FH developer Calico Pie. See the first few Popular Topics in Knowledge Base > Glossary of Terms.

Re: TNG sync

Posted: 18 Apr 2016 21:50
by cjdenbow
tatewise wrote:Carl, I think you misunderstand who we 'guys' are. This is the Family Historian User Group (FHUG), that is run by volunteers completely independently from the FH developer Calico Pie. See the first few Popular Topics in Knowledge Base > Glossary of Terms.
I suspect that Calico Pie folks monitor these posts. I guess I should have said, "those guys." I understand your criticism of my wording.

Re: TNG sync

Posted: 08 May 2016 17:51
by tatewise