* Issue with Export Gedcom plugin v5.2

[kempons]

Hi all,

I have encountered an issue when trying to export my gedcom for updating my TNG website. The last time I successfully did this was 31/12/2022 (New Years Eve).

Since then I have been updating records, merging duplicates, cleaning places, added some media and just general data entry.

The only difference I am aware of since the last successful export is the Export Gedcom Plugin has been updated to v5.2 (from v5.1).

Issue encountered is that after exporting 49% of my database, FH stops responding and never comes back. I have left it for 10 minutes and 4 hours and the same issue. Memory and CPU change throughout, but FH is not responding and needs to be hard reset.

The record(s) in question when FH hangs, have not been updated for 3 years.

Any ideas please? My users are requesting the latest updates on the website

Thanks

[tatewise]

Sorry about that Andrew.
It looks like the Individual record with Rec Id 373534 or perhaps the next Individual record is causing the problem.

I trust that you have checked the Extra Options and Other Options tabs have your preferred settings.
Also that you have run the recommended tasks before running the plugin to review the updated data.
i.e. File > Validate... and Tools > Plugins > UDF List, etc.

It is a very large Project, with over 450K Individuals, so it may be pushing the FH Lua 32-bit application to its limits.
I have tested the plugin against Projects with over 90K Individuals, but yours is 5 times larger.

In order to eliminate the changes added in Version 5.2 please try running the Version 5.1 plugin Zip file attached.
It is a ZIP file because the raw Plugin is bigger than the Forum attachments limit.
So you must right-click the downloaded ZIP file and use Extract All...
Then double-click the extracted FH Plugin (Lua) Export Gedcom File - V5-1.fh_lua file to auto-install it into FH.
If the Version 5.1 plugin has a similar problem then some new data structure in your Project is upsetting the plugin logic.

[kempons]

Hi Mike,

Thanks for the tips and for supplying Export Gedcom 5.1 plugin. I tried the 5.1 plugin and had exactly the same result.

Reviewing the record reported when FH hung and the next record found the issue. The following record had an excessively large and superflous note which I removed. Once removed, 5.2 export worked perfectly.

Apologies for the miss-direction to the plugin as the cause. The issue was the data within my project.

Thanks again for your help.

[tatewise]

No problem. Even a large Note should not cause the plugin to hang, but nothing is perfect. Glad you resolved the problem.

[tatewise]

Andrew, do you have a Project Backup that contains the excessively large Note field that you could restore?
Alternatively, can you recreate a similar excessively large Note field in your Project?

I think I have suitably revised the plugin to handle such large Notes more efficiently and use fewer memory resources.

If you are able to test my theory then try the Export Gedcom File plugin attached to the GEDCOM to Ancestry (21543) thread.

Let me know how it goes.

[kempons]

Mike,
I restored my Project backup from before I reported the issue and ran the new Export Gedcom plugin you provided.
Unfortunately I experienced exactly the same issue on the same record as I reported.

When I removed the excessive note, then Export Gedcom worked successfully as expected.

Sorry that I could not help further. I will continue with my work around of having the note deleted in my master Project.

[tatewise]

Could you attach a copy of the Note text or at least give me some details about its size and content so that I can try and recreate the symptoms.

[kempons]

Hi Mike,

Here you go. As I said, superflous and extraneous which is why it is now deleted from my main project file.

Still cleaning my FH project after switching 3+ years ago

[tatewise]

Thanks for that Andrew. I just need to check what I'm seeing.
When I unpack and open that Note text in Windows Notepad it seems to be all on one very long line of 6 million plus characters and mostly web URLs.
Is that correct or have the newline characters gone missing?

[kempons]

That is correct Mike. It is one great big single line of text with no newline characters.

[tatewise]

Andrew, I've confirmed that it is the very long Note text that upsets the plugin.
It does not matter whether it is one long line or multiple lines. In this case size matters!
The plugin has to perform various processes on Note text, e.g. Rich Text conversion, etc.
It joins all the CONT/CONC GEDCOM lines into one long Note string and subsequently split it back to CONT/CONC lines.
Lua just cannot cope with the processing of such long text strings and runs out of resources.

Therefore, I've added some extra traps for handling and reporting such excessively long Notes.

Try the attached Export Gedcom File plugin Version 5.2.2 Date 15 Feb 2023 Zip file.
It is a ZIP file because the raw Plugin is bigger than the Forum attachments limit.
So you must right-click the downloaded ZIP file and use Extract All...
Then double-click the extracted FH Plugin (Lua) Export Gedcom File.fh_lua file to auto-install it into FH.

It should proceed a bit quicker, report the excessively long Note text, and run to completion (I hope).

[kempons]

Mike,

Good news, the plugin does in fact report upon the very long note. It then allowed me to acknoweldge the report of the long note and continued to completion of the export.

Thank you for your persistance in this issue and providing a fix.

[tatewise]

Thank you for the confirmation Andrew. That fix will be in the next published version in the Plugin Store.

Interestingly, FH does not allow a Note greater than 500,000 characters to be created.
Yet, it allows your 6 million character Note to be displayed and File > Validate... does not complain!