* New version of loadrequire snippet for FH5/6

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
Post Reply
User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 25 Jan 2021 16:56

The package for the uf8 library for FH6 includes a subdirectory structure that the current version of loadrequire doesn't support.

I've created a modified version that supports this -- it works for me but I'd like others to test it before I update the snippet.

Code: Select all

require 'lfs'
require 'luacom'

local function httpRequest(url)
  local http = luacom.CreateObject("winhttp.winhttprequest.5.1")
  http:Open("GET",url,false)
  http:Send()
  http:WaitForResponse(30)
  return http
end -- local function httpRequest


function loadrequire(module,extended)
  if not(extended) then extended = module end
  local function installmodule(module,filename,size)
    local function makesubdirs(path)
      local sep, pStr = package.config:sub(1, 1), ""
      for dir in path:gmatch("[^" .. sep .. "]+") do
        pStr = pStr .. dir .. sep
        lfs.mkdir(pStr)
      end        
    end

    local bmodule = false
    if not(filename) then
      filename = module..'.mod'
      bmodule = true
    end
    local storein = fhGetContextInfo('CI_APP_DATA_FOLDER')..'\\Plugins\\'
    -- Check if subdirectory needed
    local path = string.match(filename, "(.-)[^/]-[^%.]+$")
    if path ~= "" then
      path = path:gsub('/','\\')
      -- Create sub-directory
      --lfs.mkdir(storein..path)
      makesubdirs(storein..path)
    end
    local attr = lfs.attributes(storein..filename)
    if attr and attr.mode == 'file' and attr.size == size then return true end
    -- Get file down and install it
    local url = "http://www.family-historian.co.uk/lnk/getpluginmodule.php?file="..filename
    local isOK, reply = pcall(httpRequest,url)
    if not isOK then
      fhMessageBox(reply.."\nLoad Require module finds the Internet inaccessible.")
      return false
    end
    local http = reply
    local status = http.StatusText
    if status == 'OK' then
      length = http:GetResponseHeader('Content-Length')
      data = http.ResponseBody
      if bmodule then
        local modlist = loadstring(http.ResponseBody)
        for x,y in pairs(modlist()) do
          if type(x) == 'number' and type(y) == 'string' then
            x = y -- revert to original 'filename' ipairs modlist
            y = 0
          end -- otherwise use ['filename']=size pairs modlist
          if not(installmodule(module,x,y)) then
            break
          end
        end
      else
        local function OpenFile(strFileName,strMode)
          local fileHandle, strError = io.open(strFileName,strMode)
          if not fileHandle then
            error("\n Unable to open file in \""..strMode.."\" mode. \n "..
              strFileName.." \n "..tostring(strError).." \n")
          end
          return fileHandle
        end -- OpenFile
        local function SaveStringToFile(strString,strFileName)
          local fileHandle = OpenFile(strFileName,"wb")
          fileHandle:write(strString)
          assert(fileHandle:close())
        end -- SaveStringToFile
        SaveStringToFile(data,storein..filename)
      end
      return true
    else
      fhMessageBox('An error occurred in Download, so please try later, or perform a manual download from the FHUG Knowledge Base')
      return false
    end
  end
  local function requiref(module)
    require(module)
  end
  local res, err = pcall(requiref,extended)
  if err then
    if  err:match("module '"..extended:gsub("(%W)","%%%1").."' not found") then
      local ans = fhMessageBox(
        'This plugin requires '..module..' support, please click OK to download and install the module, which may take a few minutes in some systems',
        'MB_OKCANCEL','MB_ICONEXCLAMATION')
      if ans ~= 'OK' then
        return false
      end
      if installmodule(module) then
        package.loaded[extended] = nil -- Reset Failed Load
        require(extended)
      else
        return false
      end
    else
      fhMessageBox('Error from require("'..module..'") command:\n'..(err or ''))
      return false
    end
  end
  return true
end
To test it, use

Code: Select all

loadrequire("compat53")
loadrequire('utf8')
utf8 = require('.utf8'):init()
compat53 consists of a single file within the plugins directory, and a single subdirectory.
utf8 consists of a single file within the plugins directory and a multi-level set of sub-directories.

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 10:25

Firstly, there is a problem with the code line:
if path ~= "" thenutf8require('.utf8'):init()

So I changed that to:
if path ~= "" then
utf8require('.utf8'):init()

which failed for utf8require(...) so I deleted utf8require('.utf8'):init() and restored to original if path ~= "" then

Then loadrequire("compat53") worked as it always did and uses local function makesubdirs(path) Ok.

But loadrequire("utf8") does absolutely nothing because require(module) returns true.
So utf8 = require(".utf8"):init() fails because there is no utf8.lua or utf8 folder.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 26 Jan 2021 10:31

Updated the snippet to remove the stray text.

Have you deleted your previous incarnations of utf8?

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 10:35

Don't treat me as an idiot. I'm running FH V6 and there is no utf8 in Plugins.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 26 Jan 2021 10:40

I'm not treating you as an idiot. I was checking that after you'd spent some time trying to get utf8 working, you'd cleaned up all the stuff that didn't work. (A bit like you always ask people have they done the obvious stuff...)

If there is no utf8 in plugins, I don't understand how "loadrequire("utf8") does absolutely nothing because require(module) returns true."

P.S. The only thing I've changed in the snippet is the handling for multiple su-directories.

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 10:50

It seems to be something to do with C:\ProgramData\Calico Pie\Family Historian\Plugins\compat53\utf8.dll
If I rename that to utf-8.dll the loadrequire("utf8") works. Why is your setup any different?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 26 Jan 2021 11:01

The short answer is: I don't know.

I'm working with a new install of FH6 (in a Windows 10 Hyper-V VM). The code I posted works properly for me (once I'd fixed the stray text that got pasted in when I posted the code here).

Compat53 has to be required before utf8 is initialised, because utf8 modifies the utf8 library within compat53 to add extra functions, but try (if you're willing)

Code: Select all

loadrequire('utf8')
loadrequire('compat53')
utf8 = require('.utf8'):init()

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 12:06

Yes, requiring utf8 before compat53 does load both packages and adds utf8 functions.

Many of the utf8 and compat53 features do work as far as I can tell but don't have time to test everything.

The one exception is that I think utf8.charpattern should be [%z\1-\x7F\xC2-\xF4][\x80-\xBF]* for Lua 5.1 and not [\0-\x7F\xC2-\xF4][\x80-\xBF]* but maybe that is no longer necessary?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 26 Jan 2021 12:30

Mike, thanks for that. As others have been using compat53 and utf8 for some years, I suspect (hope) most bugs have been worked out...

utf8.charpattern comes from the Lua 5.3 utf8 library (backported as part of compat53), so I think we have to leave it as documented.

I'll update the KB to include:
  • How to loadrequire the libraries
  • Mention the availability of the utf8 libraries in 'Understanding Lua Patterns'
  • Edit the 'Unicode String Functions' snippet to mention the availability of the utf8 libraries.
  • Edit 'Writing Plugins Compatible with Versions 5,6 & 7 to mention compat53
Do you think we should be recommending the utf8 libraries rather than the code snippet?

Also, a while back Valkrider asked that the compat53 and utf8 libraries be included in Install Library Modules for V6 -- is that something you can add to your expanding to-do list?

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 13:58

Am I right in thinking that the utf8 library module is also needed for FH V7?
If so, how is it loaded, or will it be included in a future update to FH V7?
I don't think that Lua 5.3 offers all those utf8 functions.

So before recommending using utf8 over anything else, does FH V7 need to support it?

The utf8 functions do replace and extend all the features of the Unicode String Functions snippet.

I can add the libraries to the Install Library Modules plugin once the FH V7 situation is a bit clearer.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 26 Jan 2021 14:02

The add-on utf8 library is included in FH7 (from 7.0.1.2), for use with the built-in 5.3 lua library.

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 14:13

OK.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 26 Jan 2021 14:55

Would you like to check the attached plugin installs the compat53 and utf8 library modules correctly.

If the modules already exist it should do nothing.

If either module is missing then it should create all the folders & files needed in the Plugins folder.

The attached script only has compat53 and utf8 because the full set of modules is too large for Forum Attachments.

[ Attachment deleted ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 28 Jan 2021 16:25

tatewise wrote:
26 Jan 2021 14:55
Would you like to check the attached plugin installs the compat53 and utf8 library modules correctly.
First run (clean FH6 install with no libraries installed in ProgramData at all)

[string "C:\Users\atira\AppData\Local\Temp\~fhD5FC.t..."]:48: attempt to index global 'zip' (a nil value)

Second run:

An error has occurred - plugin failed to complete
[string "C:\Users\atira\AppData\Local\Temp\~fhD5FC.t..."]:41: module 'compat53' not found:
no field package.preload['compat53']
no file 'C:\ProgramData\Calico Pie\Family Historian\Plugins\compat53.fh_lua'
no file 'C:\ProgramData\Calico Pie\Family Historian\Plugins\compat53.lua'
no file 'C:\Program Files (x86)\Family Historian\Program\compat5351.dll'
no file 'C:\Program Files (x86)\Family Historian\Program\compat53.dll'
no file 'C:\ProgramData\Calico Pie\Family Historian\Plugins\compat53.dll'
No changes have been made to data records.

compat53.zip is present in the target directory, and contains the right set of files, but it hasn't been unzipped.

Valkrider hasn't got it to work either.

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 28 Jan 2021 17:10

Sorry, I should have given clearer instructions.
The plugin relies on the zip.dll library module to unzip the ZIP files.
I had expected you would run my subset plugin that only loads compat53 and utf8 with all the other libraries present.

The attached variant of my subset plugin does ensure the zip.dll library exists and loads it if necessary.

Please start from a clean FH V6 install or delete the compat53.zip or utf8.zip files.

[ Attachment deleted ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
Valkrider
Megastar
Posts: 1534
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by Valkrider » 28 Jan 2021 17:28

Sorry @Mike still not working in Crossover with V6 of FH
subset.png
subset.png (24.65 KiB) Viewed 3142 times

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 28 Jan 2021 17:34

Colin, are you sure you deleted the compat53.zip file from the Plugins folder.
If that zip file exists then that error message is the symptom.
Make sure there are no compat53 or utf8 files or folders in Plugins.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
Valkrider
Megastar
Posts: 1534
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by Valkrider » 28 Jan 2021 17:44

Thanks @Mike I missed one of the files. Once I deleted it it ran fine and installed the folders and the main files.

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 17 Feb 2021 11:34

Is everybody happy for my Install Library Modules Plugin to be re-published with the compat53 and utf8 libraries included?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 4854
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by ColeValleyGirl » 17 Feb 2021 11:50

I'm happy, Mike.

User avatar
tatewise
Megastar
Posts: 27088
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New version of loadrequire snippet for FH5/6

Post by tatewise » 18 Feb 2021 16:01

The Install Library Modules Plugin Version 1.1 Date 18 Feb 2021 is in the Plugin Store.
Who can spot my deliberate mistake in the screenshot?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

Post Reply