* Export Plugin Problem FTM2019==>FH7==>TNG

[tokyojef]

Mike,
Thank you for your kindness

I have looked in my main and external drives, and nothing left of the FH6 projects.
Only the upgrade FH6 .exe with a key that no longer works.

All my files are on Ancestry, as well as backed up in externals and in the cloud.
PERHAPS in the cloud, I MAY have saved something of FH6 files... not sure. I will need to check.

If I have the files, I still need to move forward to FH7,
if we can get the export plugin working, which I would rather do.
If my FH7 expires, I can always purchase it when you get the plugin working?

Thanks again Mike!
Jeff

[tatewise]

I have asked CP for the circumstances that cause the fhConvertUTF8toANSI function to report 'String conversion failed'.

I am working on some further diagnostic plugins that may give an insight into the problem.

if you use Tools > External File Links... are there any Broken Links or any other unusual entries?

Presumably, all the Media image files display correctly within FH on the various Media tabs and the Media Window.

[tokyojef]

Hi Mike,
These are the usuals:
Check for Unlinked media OK
UDF List OK

I have screenshots of :
5 (External File Links window)
6 (proj. stats - Records)
7 (proj. stats - Facts)
8 - Conflicting place record names (in the next post)

37 external files are missing (searched 93,109 folders)
I probably can find them in an online tree, or in the cloud.

Let me know whats next when you have a chance,

Jeff

5 (External File Links window).jpg (369.96 KiB) Viewed 1102 times

6 (proj. stats - Records).jpg (285.1 KiB) Viewed 1102 times

7 (proj. stats - Facts).jpg (366.28 KiB) Viewed 1102 times

[tokyojef]

8 - Conflicting place record names:

8 - Conflicting place record names.jpg (452.92 KiB) Viewed 1105 times

[tokyojef]

Hi Mike,
The external missing 37 files are evidently "notes" that open in a firefox .htm

I am in the process of converting each one into .jpg

then drop them in the media folder and see if that helps?

Jeff

[tokyojef]

Mike,

OK!
I had to do 2 things.

1. external broken links seem to be important.
a. I changed those .htm notes into .jpg's in the media file.
b. I made sure the .jpg and previous .htm names matched.
c. the names in the broken links window showed truncated names,
so I had to edit and rename them to match the ones in the media file.
d. after that the external link check show "0" problems.

2. I again encountered the ansi/utf-8 issue pop-up that stopped the export plugin from continuing.
MY SOLUTION:
a. EACH TIME
1. reopen FH and do a seach for the filename that was causing the issue, in the properties box - rename the title
(remove the accent character, replace it with a english alphabet letter), and adjust it's file location name as well.
2. In the media folder, remove the accent mark and replace it with a regular letter as well.
b. CRASH, REPEAT & RINSE until FINALLY the plugin made it clean thru and reported the good news that the gedom was created!

Jeff

[tatewise]

Jeff, I explicitly asked you NOT to change the faulty file names, because we may not now be able to find and fix the fault!
Can you please put the accented letters back and check that the fault gets reported.

I have not yet had a reply from CP to explain what causes the error message.

I have written a very simple plugin that loops through each Media record and invokes the fhConvertUTF8toANSI function on each File Link path string similar to the Export Gedcom File plugin.
On completion, it produces a Result Set listing each Media Record, its File Path and Error Status for the first 1000 files.
So please download and run the attached fhConvertUTF8toANSI Test plugin Version 0.1 Date 14 Jan 2023.
How many File Path string conversions fail and how many have no error?
If only some fail is there anything that differentiates them from those that have no error?

I've attached an updated Export Gedcom File plugin Version 5.1.5 Date 14 Jan 2023 ZIP file.
It is a ZIP file because the raw Plugin is bigger than the Forum attachments limit.
So you must right-click the downloaded ZIP file and use Extract All...
Then double-click the extracted FH Plugin (Lua) Export Gedcom File.fh_lua file to auto-install it into FH.

This version does not produce popup error messages but adds error messages to the Result Set displayed on completion.
The error messages will be the first 100 lines with the following format:
0 -4 Media Object Media Conversion ( TNG ) 0 @...@ XXXX nn fhConvertUTF8toANSI - String conversion failed <File Path>

Are the File Paths that report errors similar to the ones reported by the fhConvertUTF8toANSI Test plugin?
The Media files that are within the Project Media folder will be prefixed by the Project Media folder path in the Export Gedcom File results.

[tokyojef]

Hi Mike,
Sorry I got excited when a workaround occured to me.

You are operating from a different pespective to get the plugin
working in all enviornments, my apologies.

I have set all back as it was.
It seem on my end,
processing French and German accent marks
always hit a wall. 2-3, so that is the differentation from what gets thru.
The folders and filepaths are all the same as original where we were.
I never changed that.

I will give the test plugin a whirl and report back.

Jeff

[tokyojef]

Mike,

The test showed no signs of errors in the first 1000.
The export plugin cleared and finished, but with errors.
The accent marks seem to trip it up.
I have included the .csv's of those for you.

Jeff

SFG2023-RM.zip

(31.03 KiB) Downloaded 24 times

[tokyojef]

Mike,
A little more clarity as to why my fixation on getting this plugin working for me==>

In an imported gedcom from FTM or RM ===> FH
or a RM tree ===> FH...the medialinks and associated medialinks are always intact.
and the interface in FTM, RM, or FH look fine.
But in TNG I lose a very large amount of medialinks and associated medialinks,
so 20% of people have the TNG import looking right.

Not sure if this is a TNG issue or something in the way the FH export plugin is working.
It seems like with the tremendous work you have put into the plugin, it should be not an issue.

Thanks again,
Jeff

[tatewise]

Contrary to your claim that "The test showed no signs of errors in the first 1000." the CSV does have one entry with the error fhConvertUTF8toANSI - String conversion failed and if you sort the Result Set on the Error Status column it is listed at the top for File Path: T:\SFG2023-RM\SFG2023-RM_media\La-Ferrière-ANC.jpg

Please run the attached fhConvertUTF8toANSI Test plugin Version 0.2 Date 15 Jan 2023 which only lists the errors so should list the 5 files reported by the Export Gedcom File plugin. Please confirm those results.

That is an extremely simple plugin that does not fail in my Projects with Media that have similar accented characters.
So something very unusual is specifically upsetting your installation of FH V7.0.18.

Regarding the Export Gedcom File issues with Media migrating to TNG, please start a new thread for that specific problem.
Please keep your original Project that has the Media filename problems.
Use the File > Project Window... and it's More Tasks... button option to Copy Project... and create a duplicate Project in which you can fix the accented character filenames and run the Export Gedcom File plugin.

[tokyojef]

Mike,

Im a little confused..

"Regarding the Export Gedcom File issues with Media migrating to TNG, please start a new thread for that specific problem."

So you are having me run the test to confirm is all; and continue the problem in another thread?
with you feeling it is not an issue of the plugin? and turning the issue over to a TNG problem?

If the plugin is failing with errors on my end with my gedcom,
it is still an issue of the plugin not being able to process the gedcom coming out of RM?
Is it not?

even when i fix the errors below in the past,
the plugin produced a gedcom that was full of errors.
This is due to a messy gedcom pre-plugin?
or that in converting to TNG, the plugin has to eliminate some things?

The gedcom format seems to have some standard practices, but program to program,
seems to alter things about it differently, or they would all play along nicely in the sandbox.

Maybe I don't have a clear enough understanding compared to your knowledge,
Wheather here or in another thread,
I am simply looking for the solution to process an RM imported Ged thru FH into TNG.
That seems to rely on your plugin working correctly for me.
Thank you for your help,
Jeff

9 -fhConvertutf8toANSI ver jan 15 results.jpg (208.75 KiB) Viewed 958 times

[tatewise]

Your results from running the fhConvertUTF8toANSI Test plugin confirm there is an unusual problem with your installation of FH v7.0.18 regarding the fhConvertUTF8toANSI function and may need CP to get involved to fix it.
Currently, I don't think it is a GEDCOM problem.

I would like this thread to focus on that specific problem with the Project that has accented characters in filenames.

When the Export Gedcom File runs successfully, you say there are problems in migrating some Media to TNG.
I would like to keep that investigation in a separate thread involving the Project without accented character filenames.
It may be caused by the way your Project uses Media or it may be you have misunderstood the instructions for uploading the exported GEDCOM and Media files to TNG. It may need assistance from other TNG users as I don't have a copy of TNG.
However, I want to avoid that discussion being mingled with the fhConvertUTF8toANSI problem.

[tokyojef]

OK Mike,
Thank you, I understand
I was careful to follow FH==>TNG instructions,
but I will discuss FH gedcom import on the TNG forums then.

You will be working with a CP on this to solve the export plugin
and the issue you see with my installation?
and hopefully let me know before my 26 days are up?

If I correct the issues in the filenames and run the plugin, from what you see..
I SHOULD be able to export it intact into TNG as long as I follow correct proceedures?

Thanks greatly Mike,

Jeff

[tatewise]

Jeff, I'm not asking you to open a thread in a TNG forum. I'm asking you to open it here in this FHUG Importing and Exploring forum, so that I and FH users who successfully migrate regularly to TNG can resolve your problem.

I may have to ask you to contact CP (Calico Pie the FH developer) so that they can investigate what is different about your installation of FH.

[tokyojef]

Ok Thanks Mike, got it.
I do want it to work for me
....in japanese yen , FH7 is between $75-100 USD.
Jeff

[tatewise]

OK.
Please run the attached fhConvertUTF8toANSI Test plugin Version 0.3 Date 15 Jan 2023 which still only lists the errors but is even simpler than before. Please confirm it lists the same 5 files.

This may form the basis of an error report to CP that we can discuss tomorrow when they are back at work.
Hopefully, the problem can be fixed within your trial period.
Please confirm what Windows version you are running.

[tokyojef]

Here you go Mike,

10 -fhConvertutf8toANSI ver jan 16 results.jpg (293.41 KiB) Viewed 918 times

--------------------------------------------------------------------

Processor Intel(R) Celeron(R) CPU 1000M @ 1.80GHz 1.80 GHz
Installed RAM 4.00 GB (3.88 GB usable)
System type 64-bit operating system, x64-based processor

Edition Windows 10 Home
Version 22H2
Installed on ‎26/‎06/‎2021
OS build 19045.2486
Experience Windows Feature Experience Pack 120.2212.4190.0

[tatewise]

OK, thank you for those details.

Could you please try one or two more experiments with the fhConvertUTF8toANSI Test plugin to determine if the problem is just confined to your RM2023FH Project or occurs with all Projects.
e.g.
Open the Family Historian Sample Project, then open the Records Window and the Media tab.
Open any Media Property Box and modify the File: link text to have a path that has an accented character.
Don't worry if the matching Media file does not exist. You do not even have to Save the changes.
Run the fhConvertUTF8toANSI Test plugin. Does it list the accented character File path with the error?

Repeat with any other Project in the same way. Does the plugin list the accented character File path with the error?

[tokyojef]

Hi Mike,
I ran the test on the sample with Ian Munro...changing the u to German umlaut ü.

11 -fhConvertutf8toANSI ver jan 17r esults SAMPLE FH file.jpg (293.38 KiB) Viewed 883 times

[tatewise]

So that confirms the problem is widespread and not confined to one Project.

I have had a reply from CP regarding this problem. The software causes they suggested have all been ruled out.
They say that since the problem only exists on your PC they are not keen to contact you to investigate the problem.
CP ask if there is anything 'unusual' about your Windows 10 PC. Is it a Japanese version or have any special settings?

I notice that you say it is Windows 10 Version 22H2 which was released late in 2022.
Yet you say it was installed on ‎26/‎06/‎2021. So did you mean Version22H1?

Can you possibly install FH V7.0.18 on another PC that you own to see if it exhibits the same problem or not.

[BillH]

Mike,

I think that date just means when Win 10 was installed. My computer shows:

image1.jpg (11.84 KiB) Viewed 837 times

I just installed 22H2 a couple months ago, but I installed Win 10 back in 2020.

Bill

[tatewise]

You are on the right track Bill and on further investigation, the Installed on date is when the Windows OS core was installed.
Mine is 30/‎04/‎2021 which is much later than when WIn 10 was first installed but is when a later version was installed.
I think 21H1, 21H2 and 22H2 all use the same Windows OS core so the installation date does not change.
Instead of Start > Settings > System > About check Start > Settings > Update & Security > Windows Update > View update history > Feature Updates and that shows when each of 21H1, 21H2, and 22H2 were installed.

[ColeValleyGirl]

It may also be worth exploring Jeff's PC's language settings...

[tatewise]

CP have now responded with some useful feedback and Martin says:

I have discovered how to reproduce the problem. If you set the system locale to Japan in the control panel, you can get the problem to occur quite easily. Calls to fhConvertUTF8toANSI appear to be more likely to fail when the locale is Japanese. Microsoft documentation is not clear about why this is. Their advice is simply to avoid ANSI.

If your user is English, you could ask him what the system locale of his PC is currently set to. To do this, don't open Settings. Instead type "Control Panel" into the search box and choose the "Control Panel App". Then click "Clock and Region". Then "Region". Finally on the 'Administrative' tab, look at "Language for non-Unicode Programs". Does it say "Japanese (Japan)"? If so, consider switching to "English (United Kingdom)". He will probably have to restart his PC. I can't guarantee that there may not be some undesirable consequences of doing this. I'm not aware of any. I would assume that he could switch back to Japanese whenever he wants to. But I can't guarantee that either. It has to be an "at-your-own-risk" type of thing. If he has concerns, I suggest he does a search on Google to make up his own mind before doing anything.

Incidentally, if you try to find the system locale via 'Settings' you can get there that way too, but it's more long-winded and along the way, the 'Region' page in Windows 10 settings may well say "United Kingdom" even if the hidden away System Locale (when you eventually find it) says 'Japanese'.

Jeff can you comment on this suggestion from CP please.

Martin goes on to ask:

Finally - why are you doing this anyway? Are you trying to generate versions of file names that are likely to safely work on all file systems? If so, it might be better to avoid ANSI conversion and try another method (e.g. writing your own 'simplify' routine which replaces all characters that aren't 'a-z', 'A-Z', '0-9' - say).

The plugin is converting file paths to ANSI in order to use the lfs library and also detects if the Conversion Loss Flag gets set so that a temporary ANSI-compatible file path can be used instead. That is the recommended way of handling file paths that may have accented characters that is compatible with FH V5, V6 & V7. What do we think of the CP suggestion?