* Lua Error in Show Project Statistics 2.6

[David2416]

Hi Mike,
I am getting this runtime error in Show Project Statistics v2.6

[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:4664: error in __gc metamethod (System error)
stack traceback:
[C]: in for iterator 'for iterator'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:4664: in function 'UpdateStatistics'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."] in function <[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:3433>
(...tail calls...)
[C]: in function 'iuplua.MainLoop'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."] in field 'ShowDialogue'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."] in function 'GUI_MainDialogue'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:4728: in main chunk

For background I am running this in a project created by importing from a Roots Magic 8.4.1 database which itself was imported from FamilySearch Family Tree. (Was curious to see what it produced!)

[tatewise]

Sorry about that David. There are a couple of possible explanations.

If you are running the Plugin in debug mode, that is an unfortunate side effect that has arisen in FH v7 with Lua v5.3.
I have reported it to CP without any significant response.

Does the Project name or any folder in its path contain accented characters or other non-ASCII characters?
Lua and Lfs are poor at handling non-ASCII characters, especially in file paths.

I am working on solutions to such deficiencies, especially now multi-language support is available in FH, so foreign language characters are more likely to appear in file paths, etc.

[Robert Jacobs]

Mike,

FWIW, I have several exhibit file names with accented characters mostly (é and è) in my media directory. They are handled perfectly by your Export Gedcom plug-in, but the file Łódź.jpg does not appear in the output GEDCOM. I ran the plug-in with the GST option.

Not too late to wish you a Happy New Year.

Robert

[David2416]

Hi Mike, thanks for your reply. I'm not running in debug (so far as I'm aware) and project name, etc is straightforward as FS-20220102d as are the folder paths. There are some accents etc in individual and place names.

I have rerun it successfully on a different project. Then rerun on this project several times successfully, followed by a failure and then OK again. There are 2685 individuals and 1232 families, 2304 places. It seems to fail only occasionally

[Mark1834]

Does the Project name or any folder in its path contain accented characters or other non-ASCII characters?
Lua and Lfs are poor at handling non-ASCII characters, especially in file paths.

Could you elaborate please? I've fallen over encoding issues a couple of times, so I now check carefully for compatibility. My understanding is that Lua and lfs library are fully ANSI compatible, so correctly handle most (but not all) European accented characters, provided that they are presented in ANSI format rather than UTF-8 (the detailed bytes are different).

Lua/lfs certainly have a problem with non-Latin languages, but you now seem to be saying that any non-ASCII characters are problematic, which I think is not correct. In my (more limited) experience, a correctly structured plugin shouldn't have issues with processing most Western European characters.

[Mark1834]

but the file Łódź.jpg does not appear in the output GEDCOM

That makes perfect sense - plugins should handle Western European ANSI characters correctly, but Łódź contains unsupported characters that are not part of ANSI, so are more of a challenge. There are other tools available that do support these, but this can raise other issues as well.

[tatewise]

Mark, I was trying to keep it simple without getting into which accented (ANSI) characters are OK and those that are not.
Since there are no accented characters at all, and the fault is only occasional, then the problem lies elsewhere, but not sure where as "error in __gc metamethod (System error)" is deep in the Lua code somewhere.

Mark, my guess is it is associated with the problem I have reported to CP, and the Directory Tree Code Snippet lfs issue discussed earlier in Knowledge Base, Libraries, Snippets, Unicode File-paths (20081). I believe I have a solution but will take time to integrate it into all my plugins.

[Mark1834]

Ok, so for the avoidance of doubt, the problem is with your plugins and your wish to keep them compatible across all FH versions. Lua/LFS does NOT have a problem with non-ASCII characters, and handles French/Spanish/German characters correctly. There are alternative solutions that are compatible will all languages, but are not compatible with FH5/6.

“Simplification” should clarify, not mislead with incorrect statements.

[tatewise]

For the avoidance of doubt, Lua & LFS do have a problem with the majority of non-ASCII characters.
Only about 127 non-ASCII (i.e. ANSI or more correctly Windows Code Page 1252) characters are supported.
I am not sure they include all the French, Spanish & German accented characters but it is possible.
They certainly do not include all European language accented characters nor the rest of the world.
The solution I plan to use employs File System Objects, works for all languages, and is compatible with FH v5 to v7.

[Mark1834]

It's probably better to talk specifics, rather than in term of vague generalities. Standard Lua supports all of the following accented and extended characters (I didn't type them all out laboriously, it's 3 lines of standard Lua to generate the list ):

€, ƒ, †, ‡, ‰, Š, Œ, Ž, ‘, ’, “, ”, •, –, —, ˜, ™, š, œ, ž, Ÿ, ¡, ¢, ¥, ¦, §, ¨, ©, ª, «, ¬, ­, ®, ¯, °, ±, ², ³, µ, ¶, ·, ¹, º, », ¼, ½, ¾, ¿, À, Á, Â, Ã, Ä, Å, Æ, Ç, È, É, Ê, Ë, Ì, Í, Î, Ï, Ð, Ñ, Ò, Ó, Ô, Õ, Ö, Ø, Ù, Ú, Û, Ü, Ý, Þ, ß, à, á, â, ã, ä, å, æ, ç, è, é, ê, ë, ì, í, î, ï, ð, ñ, ò, ó, ô, õ, ö, ÷, ø, ù, ú, û, ü, ý, þ, ÿ

If these are sufficient for your needs, there is a lot to be said for using standard Lua/lfs in your plugins. They work in all versions of FH that support plugins, are reasonably well documented (including in the plugin help as well), and there are plenty of examples and help available online. If you want more than that, a combination of FH built in functions and fhFileUtils() suppports virtually all global languages. They are FH7 only, and have one or two gaps in their capabilities, but are again reasonably well documented, easy to use, and supported by CP.

There is a very niche requirement for a handful of authors to support legacy plugins that may need to process file and path names containing other characters in all versions of FH, but IMO it is important to distinguish between what these folk may choose as their bespoke solutions and what is appropriate guidance for everybody else in new plugins.

[ColeValleyGirl]

That's what standard Lua supports if you have one particular code page loaded on your PC... so a plugin written to expect that code page won't work for users with other code pages loaded (for example, perhaps, a Dutch user who needs different characters like ij), or a Polish user who needs ł).

It also may not meet the needs of people with one code page who occasionally needs others, because their ancestors came from e.g. Greece.

[Mark1834]

Good amplification, but I think we’re on the same page (if you’ll pardon the accidental pun). The well-documented simple vanilla stuff probably supports most user requirements most of the time, but there will be exceptions that need alternative solutions.

The challenge for all of us is to avoid rival solutions that battle for attention to solve the same problem...

[tatewise]

The attached prototype Show Project Statistics plugin Version 2.6.1 Date 07 Jan 2022 has several new features.

1. The metamethod system error reported originally by David in this thread on Monday should no longer occur.
2. The Export CSV Files disrupted Show Project Statistics ~ Data.csv format reported by jelv in the thread Show Project Statistics discussion (20133) on Wednesday is fixed.
3. Based on an idea I've been developing, the plugin now handles files with any UTF characters anywhere in the path.
   That allows non-English users to name folders & files in their native tongue, whether Norwegian, Greek, Cyrillic, etc.
   The plugin uses the following methods that I'm happy to discuss with other plugin authors but in a separate thread:
   1. Uses fhFileUtils library File System Object methods for files but with extra features missing from that library.
   2. Other Lua References and Library Modules such as LFS, IM & Penlight only allow ANSI characters in filenames.
      So I have devised a scheme to let them handle files with any UTF characters anywhere in the path.
   3. Not only do these methods work with FH V7, but I believe they also work with earlier FH versions.

The purpose of attaching the plugin here is twofold:

• Firstly to check the problems reported have been fixed to the user's satisfaction.
• Secondly to allow the UTF filename features to be explored in other user systems.

[jelv]

1. The Export CSV Files disrupted Show Project Statistics ~ Data.csv format reported by jelv in the thread

[David2416]

Hi Mike,

Thank you for working on this, I have downloaded the update and will let you know how I get on. A few test runs have worked fine with no glitches.

I have noticed a slight delay (2 secs) between clicking Update Statistics and the progress bar appearing and again while repainting the Options tab. The Exceptions Detected: 1808 and Reported: 18 appear first, then the full list of check boxes 3 secs later.

Similarly between clicking Close & Report and the Result Set appearing, about 2 secs. Close & Report changes colour when mouse hovering over but does not change after clicking so a little unclear that anything happening,

Computer is quite high powered and the file contains 2909 Individuals and 746 Families. It's not a big issue but could the mouse cursor change to Hourglass?

[tatewise]

David, thank you for that feedback.
I ran some tests on even larger Projects with even more Exceptions and discovered even worse time delays.
They are caused by the techniques I'd introduced to handle files with any UTF characters anywhere in the path.
I have modified the techniques to still handle UTF file paths but without those time delays.

Please try the revised Show Project Statistics plugin Version 2.6.2 Date 10 Jan 2022 that is attached.

BTW:
If the number of Exceptions Reported is very large, there is a time delay, but that is in FH after the Plugin has closed.
It is due to the way FH compiles the large Result Set, and I think will display the rotating blue ring 'busy' icon.
There is nothing the Plugin can do in that case, not even change the cursor, because it is no longer in control.

[David2416]

Hi Mike,

Have downloaded and run against my main project. Looking good. Also ran against another project with about 30,000 people and similar number of families. It took about 1 minute to run. The result set took moments to appear with 170,180 records in result set!!!! That doesn't surprise me in the least as the project was created from an extract from FamilySearch Family Tree using Roots Magic 7 going back 40 generations. Much of the early stuff in that tree is in my opinion sheer fantasy and has loads of inconsistencies. But it serves to give large resultsets

Show Project Statistics 2.6.2 performed well. Thank you.

[tatewise]

Thank you, that sounds excellent.
If you have the interest then you could introduce some extended Unicode characters into a dummy Project name or even a few Media filenames with Face Frames, i.e. choose some exotic Greek, Cyrillic, or strange symbols.

[David2416]

OK, I'll give that a go and report back.

[David2416]

Hi Mike,

I created a project called ɖʗɲɣʘtest.fh_proj using my large Family Search derived tree. The full folder path is

C:\Users\dacoo\OneDrive\Documents\Family Historian Projects\ɖʗɲɣʘtest\ɖʗɲɣʘtest.fh_data

There are various names such as
Brian Ardcenn /mac Muiredaig Ó Dúnlainge/ born in Cúige Laighean, Ireland
Brunda Bjálki /Ögmundsson/
Erchembaldus II /von Elsaß/

Show Project Statistics 2.6.2 processed the file in about 1 minute and the resultset appeared quite quickly. Very similar to what I reported in my previous post.

Are those characters sufficent for this test? I will add some media and face frames later.

[tatewise]

That is perfect. Thank you David. That proves plugin version 2.6.2 handles those Unicode characters.
Those characters are more than sufficient for the test and there is no need to add Face Frames because the Unicode Project path will also apply to all Media within the Project.
The published Show Project Statistics version 2.6 would crash in flames with such a Project name!

[David2416]

Thank you.