* Uploading GEDCOM files

[erowland]

I recently had my aunt take a DNA test with FTDNA. In conjunction with this I set about uploading a GEDCOM from Ancestry.Com to FTDNA. I followed the instructions from Ancestry.Com on how to download my tree from Ancestry as I have done in the past to Gedmatch et al . On this occasion the downloaded Rowland Family Tree which was downloaded/saved was the Family Historian GEDCOM. I repeated the process several times and got the same result. I appreciate that this is strictly not a FamilyHistorian problem but can you suggest why this is happening?
Once again I am most appreciative of your help
Edward

[tatewise]

You are misunderstanding the Windows File Explorer display.
Every GEDCOM .ged file needs to be associated with an application that can open it.
Family Historian is the installed application on your PC that is associated with .ged files.

Usually .docx files are associated with the Microsoft Word application.
Similarly .pdf files are associated with the Adobe Acrobat Reader application.

It does not necessarily mean that those types of files were created by those applications, but simply that those applications are designated to open such files on this PC.
e.g.
Family Historian can create .pdf files from Diagrams & Reports but they are opened by Adobe Acrobat Reader.

So Family Historian will open that GEDCOM file downloaded from Ancestry.
It would be the same for any GEDCOM created by any other product.
That is how family trees are meant to be transferred between products.

[erowland]

Mike
Thank you for this clarification. Is my interpretation correct that the application used by the recipient of the Ancestry.com GEDCOM will open the GEDCOM?
BTW and for your interest, I copy below message from FTDNA regarding problems with GEDCOMs.

"Thanks for your reply. Some customers have come up with a solution to many GEDCOM upload issues.
Until we come up with a more permanent solution, I recommend you use this tool.
Please let me know if this helps. If not, then I will bug report the issue on your behalf."
https://firebric.se/fixgedcom

Many GEDCOM export files uploaded to FTDNA results in a broken family tree, where only the main person is visible in the resulting tree, or names are missing.
This tool attempts to fix these problems by adjusting the GEDCOM file to a format more suitable for FTDNA's import. Many thanks to Johanna Lindh and David Hegethorn who researched the problem and proposed a solution.
The tool attempts to

• fix id:s with leading zeros
• fix id:s that contain a dash
• fix wrong line order where TYPE appears before NAME
• extract GIVN, SURN and NSFX tags where only NAME tag is present
• capitalize names
• remove characters that FTDNA doesn't handle well in the NAME tag
• adjust date formats such as "2000-10-17", "17 Okt 2000" and "17 October 2000" into "17 OCT 2000". Any other formats are left untouched.
• fix id:s that don't contain a letter (workaround for a bug that only occurs sometimes)
• remove program specific tags, which start with an underscore
• remove characters that FTDNA doesn't handle well in notes
• remove characters that FTDNA doesn't handle well in place names

Click the button to the left and choose your GEDCOM file from your computer. It will be processed in your browser, so it will never leave your computer (and will never be saved on our servers), and the result will be provided as a downloaded file named 'fixed.ged'.
Limitations: This tool does not handle names with the character "/" well - it merely removes that character. This tool does not handle more than one million people in one tree.

Your solution to fix the IE icon problem worked perfectively

Kind regards Edward

[tatewise]

Yes, any recipient application designed to handle GEDCOM files should be able to open any GEDCOM file.
However, there are many optional GEDCOM characteristics and most applications do not support them all.
FH has a Plugin called Export Gedcom File that adjusts the characteristics of the FH GEDCOM file and exports a GEDCOM file that is compatible with many popular products including Ancestry and FTDNA.

The fixgedcomtool is similarly trying to cope with the various GEDCOM characteristics from other products and adjust them for FTDNA.

Ideally, every application should understand any GEDCOM file, but the reality is less than perfect.

[Billread]

Hi
I have read through old Topics and the latest regarding the problems uploading Gedcoms to Ancestry. I noted the suggestion previously (2007) to use IE extension Tab for Mozilla, which I use mainly, is that still necessary, as I tried instal that and couldnt access a further required add on?

But I have the same problem whatever browser I use, Ancestry takes the ged name, says it has uploaded, and reports ok. Then when I go to set myself as the root on Ancestry it states there are no people in the upload. I have tried umpteen times from Mozilla, Chrome, and IE. No luck.

SO I thought I would use a small simple ged reader to open the copy of my GEd that I have cleaned in Family Historian ( having removed all data except what i wanted, including notes in FH). Then save as, and hopefully solve the problem.

Neither My cleaned geds, or any of my main 15 current geds will open in either simple family tree, where it states ' File doesnt start with 0 head,' or in another simple tree viewer, Viewged, which reports other problems. Could this be related to Family Historian itself and how its saving geds. Because I have opened them in a Text viewer, and they all start 0 head

Now I have used these two programs over the years for simple checking of geds, and in extracting name lists and so on, which i use with my own family history website. And give to people who want a tree, but just need a simple program to use. I have never had issues before. All geds which will not open in them do open in FH ok.

So I started going back through my Geds archive, as I keep older copies each time i update a ged. BAck in 2012-2014 nearly all older copies of geds open in these two simple gedcom programs. Just the odd one with the 0 head error, But all my geds since will not open, especially any I have worked on over the last few years.

So looking at the posts on here I noticed Firebric, and their ged Fix, I went there and experimented with the Fix tool, that seems to do something to the problem geds in regard to opening with Simple FAmily Tree, or Viewged, ANd they still open ok in FH. SO any ideas why there is this problem with all my latest FH geds?

In the meantime I will try to upload a 'FIXED@ ged to Ancestry and see what happens.

help appreciated
Bill

[Billread]

Hi again
ok, a fixed ged uploaded first time using Mozilla, with no problems that I can see. So do I now have to 'FIX' all of my geds using the Firbric online tool, or is there a Family Historian Plugin that works the same way?
Lastly, WHY are all my geds giving these errors, when they are the same ones I have used over years of using FH?
Many Thanks]
Bill

[jbtapscott]

Are you using the FH "Export Gedcom File" plugin?

[tatewise]

Bill, the GEDCOM file within the FH V6 Project uses the Unicode UTF-16 character encoding by default, which is perfectly valid and saves & loads faster than the alternative option of UTF-8.
BUT, many other products do not recognise UTF-16 (including FH V5), and some do not even recognise UTF-8.
So you must convert the GEDCOM file using one of several methods:

• Use Tools > Preferences > File Load/Save and tick Save in UTF-8 file format
• Use File > Import/Export > Export > GEDCOM File whose Format defaults to UTF-8 but uses FH dialect GEDCOM
• Run the Export Gedcom File Plugin that has options for many popular products (including Ancestry) to produce a dialect of GEDCOM specifically to suit the target product.

Furthermore, for online trees such as Ancestry and FindMyPast you must take care not to upload the details of living people without there express consent. So the GEDCOM file needs to be purged of sensitive data as explained in how_to:exporting_gedcom_with_multimedia#export_to_website_without_sensitive_data|> Export to Website Without Sensitive Data.

[Billread]

Hi Mike

Just a tad confused.
so I can see I have to use a different means to get a file for Ancestry by what you say, in a different format.

So is that the same thing happening with the SImple GEd and Viewged programs as well? And if I export a version to use for Ancestry, or simpleged will it then not be in the correct version for Family Historian to function best?

Then why if I use the Gedfix online as per one of the prior posts, do the geds then open ok in these programs? Has it changed the format or fixed a problem?

I assume the plugin Export Gedcom File does what you say?
regards
Bill

[tatewise]

None of the options I listed prevent FH from working perfectly OK.

The two Export options do what they say on the tin.
They both EXPORT a GEDCOM file to a folder, leaving the FH Project GEDCOM file untouched.

It is possible that the other tools you used converted the GEDCOM file from UTF-16 to UTF-8 format.
Without investigating each one I cannot be sure.

When did you upgrade from FH V5 to FH V6?
That was when the Project GEDCOM format changed from ANSI to UTF-16 to support all the Unicode accented characters, Asian language characters, symbols, etc, etc.
So all your old FH V5 GEDCOM files would have been acceptable to Ancestry, et al, until you upgraded to V6.

However, there are other GEDCOM dialect differences between products that only the Plugin handles.
e.g. Ancestry prefers Census records to be cited by Residence facts rather than Census events, so the Plugin converts them, but only in the EXPORTED GEDCOM.

[Billread]

Hi
just in case , I have played around with the geds some now. I still am not clear what the Firebric APPlication is Fixing, but it does seem to work. So if its not changing the Standard Fh gedcom into the older UTF 8 you speak of, does anyone know what it is actually changing to make my geds readable by not only ancestry , but the Simpleged program as well?

HAve tried the gedcom export plugin as well, that also works, but it would be nice to know whats going on with geds via Firebric

I was aware of removing living persons thanks Mike
many thanks
Bill

[Billread]

PS irebric is the link at the start of this thread, that seems to have been recommenede by this post?
Bill

[tatewise]

https://firebric.se/fixgedcom is only recommended by FTDNA users in the context of upload to the FTDNA website.
You need to take care to read what is actually said and not just assume recommended applies everywhere.

I have tried firebric with an FH GEDCOM encoded in UTF-16 as described below.

The https://firebric.se/fixgedcom webpage says "It expects the file to be encoded with UTF-8". It lies!
Despite that, it did convert the FH GEDCOM encoding from UTF-16 to UTF-8 that Ancestry likes.
BUT it did not change the 1 CHAR UNICODE tag line to 1 CHAR UTF-8 which is a glaring mistake.
It did change many other tag lines that resulted in illegal GEDCOM data that may not get imported to other products.
However, that would be of little concern for FTDNA that only imports Names, Birth & Death facts and ignores the rest.

If you want to upload to FTDNA then use the Export Gedcom File Plugin in (FTD) Family Tree DNA mode.

[Billread]

I have looked myself, but cannot figure what FIrebric has actually done, I have used it on several Geds which wouldnt open in Simple Family Tree, and after 'FIXED' are fine, but reading at the start of this post what it does doesnt suggest it changes the whole format?
I hate not knowing wjhats going on
Bill

[mjashby]

Bill,

Probably the easiest way to find what the Firebric (or any other software) changes in a GEDCOM, i.e. a 'pure' text file, would be to compare the before and after versions using one of the many free text comparison software tools available, e.g. https://winmerge.org. But I would suggest starting with some relatively small files until you understand how the differences are identified.

Mervyn

[tatewise]

Bill, you may be misunderstanding what UTF-16 and UTF-8 and ANSI character encoding means.

Changing those encodings does affect the entire GEDCOM file (or any other file) but perhaps not in the way you think.
On its own, changing the encoding does NOT alter the character by character GEDCOM tags & data.
Exactly the same information is recorded in exactly the same format regardless of the character encoding.
If you printed the GEDCOM file it would look exactly the same.

What the character encoding does is store each character using a different pattern and number of bits.
e.g.

• ANSI encoding uses one 8-bit byte, so only supports 256 characters 0-9, A-Z, a-z, etc, which is quite restricting.
• UTF-16 encoding uses one or two 16-bit words, and supports 1,112,064 Unicode characters to uniquely represent worldwide language alphabets and symbols.
  The popular characters 0-9, A-Z, a-z, etc, have the same lower 7 bits in both ANSI and UTF-16.
• UTF-8 encoding uses up to four 8-bit bytes, and supports the same 1,112,064 Unicode characters as UTF-16.
  The popular characters 0-9, A-Z, a-z, etc, have the same one byte code in both ANSI and UTF-8.

Another character encoding is Morse Code but with a very limited character set.

So changing the character encoding does NOT change the character-based information, but may prevent the characters being recognised by some software (such as Ancestry).

In its defence, the https://firebric.se/fixgedcom webpage does list changes that it makes to GEDCOM files, such as fix id:s with leading zeros or that contain a dash, etc, etc...

[Billread]

Ok, thanks for that, I'll try and learn a bit more.
Bill

[Billread]

HI again
sorry to be a pain!

I have been reading back over the Gedcom issue, as I has a similar problem again the last couple of Days with a friend who sends me Geds, ( from an old Family Tree Maker), I open and use/read them in FH just by clicking on the gedcom itself. Then just save it again, as I wish to do what we always have which is exchange Geds to and fro as a means of information exchange. ( with my cousin as I got her into using FH we both worked on the same geds to and fro between us).

So if i check a Ged sent me it opens in Simpleged etc, I then open by clicking on the GED icon, but as soon as I save it from FH it no longer opens in the FTM or in simple Ged again...even though I have NOT actually imported the ged as a project in FH.

If I understood correctly the earlier comments made in your replies, FH uses your Unicode UTF-16 by default IF you import a GEd as an FH project. BUT I have never imported projects, always since the earliest Fh I had we just saved them to a folder location, exchanged them, opened them ok in programs like FTM and simpleged when sent between people who didnt use FH, and so I expected them to save as usual and be readable in these other programs.

Now though I open an FTM ged received from another relative, which is in UTF-8? similar for simpleged. But as soon as I close it in FH ( it has not been imported as a project) it becomes unopenable in anything except FH. And I have to use export as UTF-8, or the person at the other end has to use Firebric before it can be opened.

So is FH converting it to your Unicode-16 even though not imported as a project?

If so Is there a way to stop that happening, as we do this reasonably regularly. I had another person tell me a cut down ged I sent them from FH was not openeable the other Day, or do I just need to remember to always export as utf-8.
thanks
Bill Read

[tatewise]

Yes Bill, it does not matter whether in a Project or a standalone GEDCOM, when FH V6 Saves it will use UTF-16 encoding.

To change that default, use Tools > Preferences > File Load/Save and tick Save in UTF-8 file format.
That will then apply henceforth to ALL Projects and GEDCOM files.

If you don't make that Preferences change then you will always need to use the Export Gedcom methods.

BUT remember that every product will adjust some GEDCOM data structures for its own purposes that may not be compatible with other products. So quite apart from UTF-16 versus UTF-8 character encoding the Export Gedcom File Plugin adjusts those data structures to suit the target product, e.g. (FTM) Family Tree Maker.

[Billread]

Mike
following last post, a week ago my relation sent me some gedcoms from his FTM. and I opened them with no problems in FH just by clicking on them. After that they wouldnt open in Simpleged etc, and i assumed had changed something in FH. I was using Firebric to sort that, but you say that causes other problems?

Yesterday and today he sent me two geds , and each twice, but they will not just open in FH as usual. Could I send you a copy of them and perhaps you will know the problem?

Re UTF-16/UTF8 have I got it right that 16 is Unicode, and 8 is Ansi?

Also I was looking for other ways to check, and came across a program called Gedcom validator, would that be any help?

I wish there was a universal program that would just solve these issues

Regards
Bill Read

[tatewise]

Bill, we all wish there was a universal solution, but unfortunately, most products do NOT fully adhere to the GEDCOM specification and that is complicated by there being two slightly different versions 5.5 and 5.5.1.

You need to go back and read my explanations of ANSI, UTF-16 and UTF-8 again.
ANSI only ever uses one 8-bit-byte per character, so only supports 256 characters and NOT Unicode.
Both UTF-16 and UTF-8 support the 1,112,064 Unicode character set.
UTF-16 uses one or two 16-bit-words per character.
UTF-8 uses one to four 8-bit-bytes per character (its one byte characters partly match ANSI).

GEDCOM Validator is good at verifying whether a GEDCOM file adheres to the specifications, but won't fix errors.

In what way will FH not open those recent GEDCOM files?
It is extremely rare that FH will not open a GEDCOM file. Are you sure they are GEDCOM files?
If you open them with Notepad, do they start something like the following where ... can be various text:
0 HEAD
1 SOUR ...
2 VERS ...
2 NAME ...
2 CORP ...
1 FILE ...ged
1 GEDC
2 VERS 5.5
2 FORM LINEAGE-LINKED
1 CHAR ...

[tatewise]

Maybe some example characters would help understand the encodings.

Code: Select all

Char	ANSI (1-byte)		UTF-8 (1 - 4 bytes)	UTF-16 (1 or 2 words = 2 or 4 bytes)
	Hex 	Dec 		Hex 	Dec 		Hex 	Dec
		
 A	 41  	 65  		 41  	 65  		0041	0065
 
 £	 A3  	163 		C2 A3	194 163		00A3	0163
 
 Σ	not available 		CE A3	206 163		03A3	0931
 

[Billread]

Hi Mike
sorry but from reading the earlier post I thought Ansi and UTF8 were the same , I understand more what you said now on reading this latest bit from you

I knew I should have paid attention many years back when I started using a BBC micro.

I just had to break in the middle of this and answering you, as a relative was checking something online, and it seems that when I had the problem uploading to Ancestry a week or so back, I had cleaned my Main Read lines ged out of eveything I wanted , leaving just main line persons, no siblings, nothing. When I tried upload it wouldnt work, so I had used Firebric to 'FIX' it. It seems firebric took a random death date of 1895 as put it as the death date for about 5 people in that cleaned GED. I just checke dthe original, and of course its fine, no such thing, so now I have to clean my GED out again, and get a copy out for Ancestry, My Heritage, and Gedmatch uploads, which export would i use that will probably work for all three?

To get totally clear now though please

You didnt say though whether( as I have done usually when get FTM Gedcoms, and open with no problems), when I just click on a gedcom and open it, then just close it whether I alter it or not, IF it changes the format from the received FTM/other format ( UTF8??) to FH using UTF 16 using unicode. ( Have I got it right that this is what Fam Historian now uses by default??, and that prior to that it was UTF-* using Ansi

as Previously you wrote ' When did you upgrade from FH V5 to FH V6?
That was when the Project GEDCOM format changed from ANSI to UTF-16 to support all the Unicode accented characters, Asian language characters, symbols, etc, etc.
So all your old FH V5 GEDCOM files would have been acceptable to Ancestry, et al, until you upgraded to V6.)'.

As several times I have done this lately, and then I cannot open it in Simpleged ( which I assume uses an earlier coding), And Mike cannot open it in FTM etc .



Also , can we make clear, I have never imported any of my geds over the last 12 years or more into FH projects( I know you advocate it should be done as it gives other functionality). I havent done that because of the ged exchanges, especially with the cousin I did lots of to and fro of geds. In that case should all of my geds have stayed in the earlier UTF-8 using ANSI? (Or was FH using Unicode on UTF-8?)



Presumably if i had imported it as a project, I would then definately have to use the Export function as UTF8 or whatever?

From one of the Geds which will not open ( which is a newer version of the same ged I had a week ago that did open, and the front ends seem the same)
0 HEAD
1 SOUR FTW
2 VERS Family Tree Maker (13.0.285)
2 NAME Family Tree Maker for Windows
2 CORP MyFamily.com, Inc.
3 ADDR 360 W 4800 N
4 CONT Provo, UT 84604
3 PHON (801) 705-7000
1 DEST FTW
1 DATE 15 MAY 2020
1 CHAR ANSI
1 SUBM Unknown
1 FILE C:\Users\Mike Tyrrell\Documents\Family Tree Maker\Tyrrell Outram Sumner Day - Excluding Living Relatives - 15.05.2020.GED
1 GEDC
2 VERS 5.5
2 FORM LINEAGE-LINKED

From the second which will not just open

0 HEAD
1 SOUR FTM
2 VERS Family Tree Maker (19.0.0.206)
2 NAME Family Tree Maker for Windows
2 CORP Ancestry.com
3 ADDR 360 W 4800 N
4 CONT Provo, UT 84604
3 PHON (801) 705-7000
1 DEST GED55
1 DATE 16 MAY 2020
1 CHAR ANSI
1 FILE C:\Users\Mike Tyrrell\Documents\Family Tree Maker\Descendants of James Day - c1829.ged
1 SUBM @SUBM@
1 GEDC
2 VERS 5.5


Thanks a lot
Bill Read

[tatewise]

Bill, have you read my posting today Sun 17th May 2020 12:53 ?
I really don't know how to say it any clearer.
The result is the same regardless of whether we are talking about Projects or GEDCOM on their own.
FH V6 WILL SAVE THE GEDCOM USING UTF-16 ENCODING unless you change its Preference setting.

As soon as you open a GEDCOM file with FH V6 it will convert the data and save it in UTF-16 format.
Even if you do nothing, FH V6 will have added Place records and after 5 minutes will AutoSave in UTF-16 format.
It also produces an Exception Log that lists all the other changes it has made when opening the GEDCOM file.
So as soon as you open the GEDCOM file, FH V6 changes lots of things and will Save them in UTF-16.
The only way to prevent that is to disable AutoSave and refuse to Save when prompted when you close FH.
Unless as I said you change the Tools > Preferences > File Load/Save setting to use UTF-8 file format.
Then you should be able to send GEDCOM back and forth without hindrance.

Back in the day, when you had FH V5, it only ever saved in ANSI format.
It would read UTF-8 GEDCOM files but any non-ANSI characters would get converted to ? marks.
So yes, you could open and close GEDCOM and send them back and forth without hindrance.

If it is small enough, you can Attach one of those GEDCOM files to your next Reply using the ATTACHMENTS tab ADD FILES option at the bottom below the message box.

[Billread]

sorry Mike, Hard day. I understand now.

so geds on is small, other is 6557kb. Have tried attach both only one is small enough, both when clicked just throw up the import ged box
thanks

Descendants of James Day - c1829.ged

(134.78 KiB) Downloaded 135 times

Bill