* Export GEDCOM Plug - Problems with File Conversion

[ADC65]

This is probably a question for Mike Tate, but any help is appreciated

When running the Export plugin with settings for TNG output, and the Media files settings on the Basic Options set for Max Height to 1000 and Max Width set to 1000, Use JPEG set UNTICKED and Apply Settings Above to All Images TICKED, the plugin chugs away successfully but after a while (about 700 - 1000 images) it starts to stutter and throws up errors similar to the following:

Memo
Process Crop Error: [string "lua5/im_image.lua"]:24: image failed to be created, insufficient memory
Args L:0 R:6375 B:0 T:3743
Image Height: 3744.0 Width: 6376.0
Multimedia Record Id: OBJE[1484] _AREA: Full-Frame
...
Export image file may not get created

Anywhere between 2 to 20 or so similar errors will be generated. Memory on my PC is not an issue, I can see what is being used and there is plenty of memory left. The files this happens on appear to be fairly random - i.e., it will fail on different files each time. It does not happen on the first few hundred files.

I'm running a Windows 10 machine, updated, and V4.4 of the plugin.

Cheers
Adrian

[tatewise]

Sorry about that problem Adrian.
I notice we have discussed TNG exporting before in Export Gedcom File Tweaks (17998).
I must apologise for not reviewing the TNG _KEYS to _TYPE conversion to see if the Note can be excluded.
That is now on my checklist of tasks to investigate.

BTW: In that discussion, you mentioned the filename prefix. Did you see the new Frame Naming options where (FOLDER) Sub-folder Name eliminates the prefix and uses folder names instead?

Presumably, this image processing problem has only just started?
Have you any idea what has changed to cause its occurrence?
Are there new media images or have you changed the Plugin settings?

It may be difficult to discover a complete solution because the error occurs deep within the image processing library module "lua5/im_image.lua" which is outside my control. Also, memory allocation is more complex than just how much RAM is in use. There is a mechanism for reserving memory that can cause insufficient memory errors even when there appears to be plenty of empty RAM.

It is possible that FH V7 might resolve the problem because it uses a later version of that library module.

The library module sometimes seems to have problems with creating non-JPEG files, so try ticking Use JPEG file format...
Also, try reducing the Max height/width setting to the default 600 value.

[ADC65]

Hi Mike,

Many thanks for the response.

In regards to the other discussion, I have experimented a little and now when the Export finishes, I just use the dialogue to open the GEDCOM file in Notepad and then do a Search-and-Replace-All to change _KEYS to _TYPE and this does exactly what I need it to do. It takes seconds and is not inconvenient so please do not prioritise anything on my behalf.

I had not seen the Frame Naming option, thank you for pointing that out, I will experiment more later and give you some feedback.

I am using FH V7, I should have pointed that out, sorry. The problem is not new, but it seems to happen more often now - but maybe because I have more media items than previously as I have added quite a few over the last year. For what it's worth, I agree with you that I think it is an issue with reserved memory within the image processing library module.

Two observations - Firstly, when I do not empty the Media output folder before running, the Export finishes successfully with no errors - it does not recreate media items as far as I can tell, so probably this leaves enough memory to process the 'problem' ones. Secondly, if I reduce the size to 600 x 600 maximum, and output as JPEG, the plugin completes successfully with no errors whether the Media folder is emptied or not. The JPEG option is fine for me, but the 600 x 600 limit makes the images too small for what I need them for.

I am happy to work around by not emptying the Media folder, as I very rarely change or replace media. I also think that this is something outside of what you are able to fix as it is the Lua module probably using memory inefficiently. But I will play around so at least we may know what is likely to cause problems.

Thanks again
Adrian

[tatewise]

Adrian, please update your Profile on the right of your postings to identify Family Historian V7 as what you use.

The Plugin handles Media 'intelligently' as explained in the Help & Advice. It checks if Media files have changed or are missing from the Export folder and exports them again. Otherwise, it leaves the exported Media alone which is much faster.
So your strategy of leaving the Export folder populated with Media is the best approach.

See the plugin Help & Advice FAQ.

[ADC65]

Adrian, please update your Profile on the right of your postings to identify Family Historian V7 as what you use.

Apologies - I have now done this.

I have run the Export a number of times with a number of Media output configurations. My ideal output size is 1200 x 1200 maximum so I went with that. It did fail a lot (50+ times out of 1500 images), but by running it a second time, without emptying the Media Folder, these images did then work - and as you mention it runs a lot faster too as it only has to process those missing images. This is fine as a workaround for me, the only real irritant is having to click through a seemingly never-ending number of failure dialogue boxes on the first pass (nothing you can do about that I know).

My only other observation is that the file size itself doesn't appear to be a problem if it is near the beginning of the process - so the module can reduce a large file successfully at first, but it begins to fail later on. My guess is the image module is not releasing memory properly between images, and the available working space becomes gradually smaller.

I have tried your new Frame Naming option to output to a sub-folder and it is excellent. Very helpful. Thank you for providing this, and indeed all your superb work you do for the FH community.

Regards
Adrian

[tatewise]

Another user had memory issues with very large Rich Text Note text and I added some collectgarbage(..) memory recovery statements that solved the problem. I will add a check-list item to review that and image library memory options to see if there are possible solutions. But not this week!

[tatewise]

Please see Export Gedcom / FH7 problem? (18417) that offers an updated plugin version with more regular collectgarbage(..) memory recovery statements. Does that solve your Media file conversion issues?

[ADC65]

Hi Mike, unfortunately this hasn't changed anything for me; I still get the same errors and probably about the same amount - about 100 errors for about 1,500 media items. Re-running the plugin creates the skipped files successfully. The same error messages are generated as previously, i.e., insufficient memory. Thanks, Adrian.

[tatewise]

Having gone back over this thread the original error message struck me as a bit odd by its reference to lua5/im_image.lua because I cannot find any such lua5 folder or im_image.lua file.
Please check your C:\ProgramData\Calico Pie\Family Historian\Plugins\ folder as follows.
It will have plenty of .fh_lua Family Historian Plugin (Lua) files.
It may have some .disabled_fh_lua DISABLED_FH_LUA files.
It should NOT have any .lua LUA files or .dll Application extension files or any folders.

[ADC65]

In answer to your question about the LUA files, this is the content of that directory:

Directory of C:\ProgramData\Calico Pie\Family Historian\Plugins

02/04/2021 23:28 <DIR> .
02/04/2021 23:28 <DIR> ..
24/03/2021 00:51 9,854 Add Media URL Shortcut.fh_lua
10/03/2021 14:30 160,345 Ancestral Sources Data Entry V5.0.fh_lua
04/10/2019 11:10 257,217 Backup and Restore Family Historian Settings.disabled_fh_lua
22/03/2021 18:52 11,059 Check for Unlinked Media.fh_lua
02/04/2021 23:24 540,162 Export Gedcom File 4.4.2.fh_lua
11/08/2020 13:46 5,373 Keyword Update.disabled_fh_lua
03/01/2021 15:43 26,961 Where Used Record Links.fh_lua
8 File(s) 1,010,971 bytes

Regards,
Adrian

[tatewise]

Please try the attached Export Gedcom File plugin Version 4.4.3 Date 03 Apr 2021 in a ZIP file.
[ EDIT: Attachment delete as in Plugin Store. ]

It does not do much to prevent the failures but it does let you inhibit the repeated popup messages.
So now you don't have to tediously click through them all.
The failures are now logged at the top of the Result Set after the plugin completes.

As a matter of interest are you running a conventional Windows 10 configuration?
It is just that the file listing you posted does not look like a Windows File Explorer listing; perhaps from some other utility.

( I have 'cleaned up' the earlier postings with 'mistakes'. )

[ADC65]

Afternoon Mike.

Yes, I'm running a standard Windows 10 machine. To get the file directory listing I just did a DIR > file.txt command in Command Prompt so I could copy and paste the results into the reply window here. It outputs in the old DOS-style format, but the results are effectively the same. If it helps I can do a screenshot.

Thanks for the updated file. I have tested it and it now suppresses errors as you advised. I ran it a few times and it appears consistent. Interestingly, it now only fails on 3 files - 2x of the "lua5/im_image.lua" errors and 1x of the File Image Load Error. When I ran the plugin and clicked "No Retain" it still only failed on three files (please note I cleared the export directory to empty before each test). Have you made any further garbage collection adjustments? I will try removing the three files it fails on (in case there are problems with the actual files) and retry later.

Regards,
Adrian.

[ADC65]

Hello Mike.

After a number of tests, the plugin seems to fail on three files. I replaced those files with similar size and dimension files, and they worked but it then fell over on three other files.

Interestingly they are at similar line numbers as logged in your results window. Running the plugin once, the fails wer at lines:

"118453"
"121060"
"125563"

Running it a second time with no file replacements, the same files failed (as may reasonably be expected):

"118453"
"121060"
"125563"

Replacing those files with similar files (same names but actually different substitution files), the plugin failed at the following lines:

"118477"
"121084"
"125663"

I don't know if this proximity /similarity is a red herring or not. Certainly something has changed from the 100+ fails I used to get.

Regards,
Adrian.

[tatewise]

I am learning more and more about memory garbage collection in FH V7 Lua 5.3 that was unnecessary in FH V6 Lua 5.1.
I admit that when a Media error arises the Plugin now performs an aggressive garbage collection not done before.
So that suggests I might be heading in the right direction.
Unfortunately, that level of garbage collection cannot be run too often because it dramatically slows the export process.
I'm working on some ideas.

The repeatably of your original files is a useful benchmark.
What are the Before and After memory Kilobyte usage counts reported in the Result Set?

You could also try the Plugin version attached to Export Gedcom / FH7 problem? (18417).

[ADC65]

I have tried V4.4.4 as requested. I now get 2 errors instead of 3. I don't see a before-and-after usage counts in the result set (I am missing where it is?), but I do get this:

"-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=16912.323242188"

I see that has a Record number attached to it ("0 @S292@ SOUR") - does that mean it has "trapped" the error at that point and performed a garbage collection?

The other two error lines are:

"0" "-1" "Media Object" "Media Conversion ( TNG )" "0 @S739@ SOUR" "119603" "1 OBJE @O801@" "Process Crop Error: [string ""lua5/im_image.lua""]:24: image failed to be created, insufficient memory Args L: 0 R: 3506 B: 0 T: 2479 Image Height: 2480.0 Width: 3507.0 Multimedia Record Id: OBJE [801] _AREA: Full-Frame Source: F:\Family Historian\ADC\ADC.fh_data\Media\Birth & Baptism\Birth Certificates\birth certificate - hendra, david.tif Export: birth certificate - hendra, david.jpeg Export image file may not get created"

"0" "-1" "Media Object" "Media Conversion ( TNG )" "0 @S1003@ SOUR" "121713" "1 OBJE @O1376@" "Process Crop Error: [string ""lua5/im_image.lua""]:24: image failed to be created, insufficient memory Args L: 0 R: 6375 B: 0 T: 3743 Image Height: 3744.0 Width: 6376.0 Multimedia Record Id: OBJE [1376] _AREA: Full-Frame Source: F:\Family Historian\ADC\ADC.fh_data\Media\Census & Location\UK Census 1911\UK Census 1911 (RG14-11780-240-2-14-225).jpg Export: UK Census 1911 (RG14-11780-240-2-14-225).jpg Export image file may not get created"

If you would like me to move this to the other thread, or continue the discussion there, that's fine.

Regards and Happy Easter,
Adrian.

[tatewise]

It seems we are moving in the right direction with reducing numbers of media failures.
I think keeping the two threads separate makes discussion of the two different symptoms easier to follow.

Sorry, I changed the Before/After report to just memory Usage where the number is in KBytes, i.e. 16 MB in your example.
I admit it is worded badly but it is simply a snapshot report of memory Usage at line 116461.
I had hoped for rather more such reports but have set the snapshot interval too high.

The other two log the failures due to insufficient memory, and trigger a very aggressive memory garbage collection.
It is not feasible to do that for every image conversion as it dramatically slows down the Plugin.

In the longer term might it be possible to reduce the size of those large images and that way avoid the memory problems?
Don't do that yet until we cannot get the Lua memory management to succeed.

Would you be prepared to carry out some experiments with the Plugin by changing the garbage collection parameters?
It involves very simple edits to the Plugin script as explained below.
Open the Tools > Plugins... window. Ensure the << Less/More >> button is showing the column of righthand buttons.
Select the Export Gedcom File plugin and click Edit... to display the script edit window.
Scroll to the very bottom where about line 9705 it says -- Main code starts here --
Just below that there are two collectgarbage functions each with a numerical parameter.
Try reducing the first below 100 and increasing the second towards or even beyond 300 perhaps by 10 at a time.
Use the File > Save command in the edit window to save the edits and then X Close to close the window.
Run the Plugin, as usual, to see if all your Media get converted. Note if the overall run time increases significantly.

Another option is to see if setting the Plugin Max height/width to 1100 to 1000 gets it over the line.

[ADC65]

Yes of course I would be happy to try all of those different parameters, I will start by reducing the output size in steps of 100 and let you know. I will note the results and timings. I will try to complete the various tests by tomorrow evening if I can. Would you like me to list them here, or send them to you directly?

I can try reducing the image sizes as one of the tests, when you want me to. I understand your thinking, but ultimately I would not want to reduce the file sizes permanently. A lot of the documents have been loaned from others and I have scanned them at a high resolution as I probably will not get opportunity to see the originals again. This is one of the reasons I find your plugin so useful - I don't have to pay for large amounts of storage with my web provider and I don't have to keep two versions of files on my computer. The output size could be smaller for me, as I offer to supply full size documents to anyone interested anyway. 1200x1200 is ideal I find for being small enough in filesize and readability.

[tatewise]

Thank you. Please post results here as it saves me having to cross-reference them.

[ADC65]

Some preliminary results. I ran the plugin using decreasing output size parameters, in steps of 100, i.e., 1200x1200, 1100x1100, ... , 600x600.

I received 2 errors at 1200x1200, 1 error at 1100x1100, 1 error at 1000x1000 and no errors at 900x900 or lower. However, at 600x600, the documents become too small and compacted to read any text on them. I have copied the memory usage figures below. The errors are at similar places and formats as previously posted.

I will start on your code amendments tomorrow and post the results as soon as I can.

1200x1200: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15732.5"

1100x1100: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15730.297851563"

1000x1000: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15704.815429688"

900x900: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15958.228515625"

800x800: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15874.280273438"

700x700: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15731.915039063"

600x600: "-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @S292@ SOUR" "116461" "Collect Garbage Memory" "Usage=15690.127929688"

[ADC65]

Mike, in terms of the two garbagecollection functions, did you want me to change the parameters independently or in tandem with each other?

[tatewise]

Try changing in tandem with the first going down 10 and the other going up 10 each time.
If necessary, depending on the results, we can try a few independent changes.
Can you try and keep an eye on run times to see if the more aggressive modes slow things down.

[tatewise]

Adrian, have you been able to glean any results from the memory garbage collection tests we discussed last week?

[ADC65]

Sorry Mike, real life got in the way! I will try my best to do it tonight/tomorrow.

[ADC65]

Hello Mike,

I have found some time this evening to run the tests and the results are below. The first set is run at the default values already in the code (100, 270). The errors stopped at (70, 300). There was no real measurable difference in the time taken to export. I have noted the times, but they include the time it takes me to restart the plugin once an error has been generated. So the time hovers around 9 minutes. I went past the point where there were no errors to see if there would be any increase in time.

Parameter 1 / Parameter 2 / Errors / Time

100 / 270 / 2 / 9:10
90 / 280 / 2 / 9:19
80 / 290 2 / 9:32
70 / 300 / 0 / 8:51
60 / 310 / 0 / 8:58
50 / 320 / 0 / 8:49

I hope this is useful. Happy to do any other testing you want me to.

Regards
Adrian

[tatewise]

That excellent and very thorough. Thank you.
It seems to be a win, win scenario.
Despite more aggressive memory garbage collection resulting in no errors, there is also no time penalty.

How is the time distributed across each of the record types: INDI, FAM, SOUR, OBJE, etc?
I suspect it is mostly OBJE image file copying.

If so, then some recent work with the Backup and Restore FH Settings plugin that also copies files may help.
Let me investigate.