* Crossover/PlayonMac/Linux winhttp download problem

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

These points refer to the Improve Website plugin error (17638) thread regarding file downloads.

I'm not convinced about Research Planner, or about setting a precedent that other plugins that download stuff need to document the downloads needed in many places, particularly as it makes moving the download files less easy. (Edit one knowledge base page versus edit more than one and you might not even know where the others are -- it's bound to break something).

The Research Planner pages within the KnowledgeBase document what's needed, and point to glossary:family_historian_program_data_folder|> Family Historian Program Data Folder. -- why not improve that page instead?

I'd much rather see some very clear generic instructions that tell the relevant users that if a plugin requires to download files to the plugin data directory or sub-directory, they should identify where the download lives from the plugin's documentation and install it.... blah blah blah. Plus some instructions for plugin writers that, if they trigger any downloads, they need to document the download source(s) for the benefit of users in non-standard environments.

But if you want to add it in, Mike, at least add all the downloads associated with RP with instructions on how to handle them. Or do the report files not cause a problem because they go direct into the custom reports directory? There may also be an issue with the queries and the fact definition file -- and that can't be fixed by documenting the download location because they're generated in code rather than being static.
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by Valkrider »

Helen

This is all on the page about running Family Historian on a Mac under Crossover or other Wine variants. These issues that the KB covers are Mac specific and I don't think are intended to be applicable for Windows users. It is probably easier (in my view anyway) to have all the Mac issues and their workarounds in one place in the KB rather than on each plugins specific pages.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

Colin,

I (think I) know which pages it's on -- the one you refer to, the page for my plugin (and also within the help file) and the Knowledge Base > Family Historian Program Data Folder page.

What I'm trying to say is it must absolutely be on my plugin page (and in the plugin help file -- for issues other than downloading the help file) and if my plugin help needs improving I'll do it. What I don't want to end up doing is having to edit multiple locations (some of which I may not know about) if I change the location that hosts my downloads, or change the name of the help file, or build in extra downloads (at least two of which are likely to happen with the next version).

So the page about running Family Historian on a Mac under Crossover or other Wine variants needs to either (a) have plugin-independent instructions that combined with the good plugin documentation allow the relevant users to get up and going; or (b) link to the plugin page to find the downloads, not direct to the download files.

There also needs to be some work done on Knowledge Base > Family Historian Program Data Folder based on the latest learnings (which is an illustration of why documenting the same thing in two different places is risky).

And guidance for plugin writers need to make them aware of this issues.

Otherwise you'll forever be running to play catch up on 'your' page as new plugins that depend on downloads are released.

Ideally, Calico Pie would modify the FH API to return the correct path in non-Windows environments as well as windows environments, and/or modify the API to allow interrogation of the operation environment so that different variants can be coded in. But I wouldn't expect that any time soon -- and certainly not so late in the FH7 development cycle. Maybe if Mike has time on his hands, he can work out a method of doing this within Lua. Penlight has a platform function which might be worth exploring...

I'm still not sure if the dynamic fact set and reports are a problem, or the static report files.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

Helen, sorry, but you have misunderstood the Wine/Mac/Linux problem.

The glossary:family_historian_program_data_folder|> Family Historian Program Data Folder page does what is intended and identifies the filing system path names used by different OS for essentially the same Program Data folder.
The Lua and FH API have no problem with finding that system path and reading, writing & deleting files in every OS.
(There used to be some problems creating folders within that system path, but not sure whether they still exist.)
The primary reason for the KB page is so humans can locate the Program Data folder when necessary.

The Wine/Mac/Linux problem is with downloading from the Internet using a script such as:
http = luacom.CreateObject("winhttp.winhttprequest.5.1")
http:Open("GET",strRequest,false)
http:Send()

because WINE does not support the winhttp command mechanism.
So library modules and files from FHUG do not download.

Therefore, files that are created within a Plugin and saved in the Program Data folder work fine.

I agree that Plugin authors must be made aware of that limitation and the need to provide a cross-reference and advice in the how_to:family_historian_v6_on_crossover_wine|> Family Historian V6 & Ancestral Sources V5 on Crossover, PlayOnLinux/Mac & Wine page, or whatever we agree is the best way to document this problem. One strategy is to encourage Plugin authors to word the download error message such that it refers to the Crossover, PlayOnLinux/Mac & Wine manual download instructions, perhaps with a code snippet example.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

In that case it's worse than I thought, and we need an alternate download mechanism as well as a way of identifying the environment to know which download mechanism to use, as a long term solution. Especially for dynamic files, where human intervention is not possible.

Is the socket library a viable alternative?

Even so, I don't think duplicating the links to plugin-specific downloads is the right way to go.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

Why worse than you thought?

Be my guest. You are welcome to investigate any alternative download mechanism.
I have spent many hours (if not days) working with Colin and others to discover the limitations of WINE/POM/POL, etc.
It is not easy without investing in the necessary WINE/POM/POL OS environment.

I have struggled with determining the OS environment, but it is extremely difficult short of a dialogue with the user.
Because FH/Lua is running in a Windows emulator it deliberately hides the Mac/Linux OS.
The only mechanism I found was the glossary:family_historian_program_data_folder|> Family Historian Program Data Folder path name, which for Crossover/PlayOnMac involves \users\public\ that is not used in native Windows paths. :idea:
But that is now invalidated as JP Ford says PlayOnLinux uses the native Windows C:\ProgramData\ path. :evil:

I've tried using the socket library for download tasks but it only supports the http protocol and not https used by most URL these days, so I got nowhere.
There is the LuaSec library from https://github.com/brunoos/luasec/wiki to support SSL sockets and https protocol, but CalicoPie could not get it to work with FH when I asked in Oct 2017 via problem report Lua Sec SSL Library for https number 123878. :?

I agree duplicating links is not ideal, so we need to agree the desired approach.
It is not rational to include library module download instructions with every Plugin that might run in a Mac/Linux OS.
So to me, it seemed logical to put the data file downloads together with the library module downloads.

This recent discussion needs to be in another discussion thread separate from the Improve Website... Plugin.
Can we pause the discussion to let me split off the recent postings?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

Worse than I thought because it's something that has to be fixed in code not sticking plaster documentation attempts that cannot handle dynamic downloads -- i.e there isn't a documentation solution for any dynamic downloads (which I suspect are going to be more viable under FH7).
It is not rational to include library module download instructions with every Plugin that might run in a Mac/Linux OS.
Of course it is -- it will be mostly boilerplate (which can be provided) and every plugin should document what is necessary to make it work in every supported environment (or state in what environments it isn't supported -- I need to update my documentation to say X and Y won't work in P & Q environments for now). Bear in mind that many people only download plugins from the plugin store and never come to the KnowledgeBase. I refer people to the Research Planner KnowledgeBase page in my Plugin Store entry and will (when I get some time) supplement that in the Plugin Store entry with advice specifically highlighting the non-windows environment limitations.

OK -- pause it. I won't be able to put in much work for the foreseeable future as I'm still working on the new knowledgebase platform.

Colin, when you and I both have time, can you help test the Penlight library solution to determining the environment? -- it might be a non-starter, but we could at least explore it.

My downloads use http so I can probably code around the problem when I have some time, but I'd rather contribute to a wider solution.
User avatar
mjashby
Megastar
Posts: 719
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by mjashby »

For anyone interested this is what currently appears in the WineHQ FAQ regarding use of winhttp which is supported, but with limitations as described:

"6.6.8 How do I configure a proxy?

If you want to use a proxy server for all HTTP connections, simply set the http_proxy environment variable. On many Linux distributions, configuring a network proxy, e.g. with the Network Proxy tool, does this for you automatically.

Alternatively, you can configure a proxy in the registry. There are separate locations for wininet.dll and winhttp.dll.

For wininet, use regedit to add the following values to the [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings] key:

"ProxyEnable"=dword:00000001
"ProxyServer"="proxy-server-address:port"
At present, Wine's wininet.dll does not support proxy autoconfiguration (PAC) scripts.

For winhttp, you'll need to use the proxycfg.exe utility to configure the registry. This utility is available in the system32 directory of a Windows installation, and MSDN describes its usage."

Mervyn
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

@Mervyn ~ Are you able to apply that winhttp advice and try one of the Plugins that need library module downloads?
But for that to be a valid test, the library module must NOT already exist in the plugins folder.
Alternatively, I could write a tiny test Plugin script that just tries to download a file using winhttp. See later...

@Helen ~ I maintain that is not rational for every Plugin that uses any problem library modules to repeat the detailed download instructions in full. It should cross-refer to the FHUG KB instructions in the download error message &/or in the Plugin Store description.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by Valkrider »

Colin, when you and I both have time, can you help test the Penlight library solution to determining the environment? -- it might be a non-starter, but we could at least explore it.
Helen

Yes no problem. Our lockdown is one of the most strict in Europe so I am not going anywhere before the middle of next month at the earliest (I suspect much longer). So any testing with this or the new KB is not a problem just let me know.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

Mervyn, I suggest the Research Planner plugin -- both installing it and the Tool tabs (which installs some dynamically created files) which should cover all the bases -- and then Mike's plugin that needs icons.

Colin, I suspect I'm stuck at home for 12 months (unless the relevant Birmingham hospital insists I come for a heart transplant assessment next month). I may contact you offline...
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

@Helen ~ I maintain that is not rational for every Plugin that uses any problem library modules to repeat the detailed download instructions in full. It should cross-refer to the FHUG KB instructions in the download error message &/or in the Plugin Store description.
We have a philosophy disconnect -- I don't want to repeat the detailed download instructions and would prefer to link to some generic ones, but I don't want to repeat the links to a specific download when those links may and will change. Note, I'm not talking about the library modules, which should and can be handled generically; I'm talking about plugin-specific downloads.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

Helen ~ You now agree with my ORIGINAL statement that only mentions library modules, not Plugin specific downloads:
It is not rational to include library module download instructions with every Plugin that might run in a Mac/Linux OS.
I said I agreed that duplicating links is not ideal, so we need to agree on the desired approach.
We are in danger of vehemently agreeing with each other not a philosophical disconnect :)
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

Thanks, Mike -- as far as I can see most of this has been about plugin-specific downloads, not libraries which can be handled generically.

Do you have a proposed way forward for coding plugin specific downloads? I'm seeing several strands:
  • http rather than https, using the luacom library
  • the proxy solution suggested by Mervyn
  • http using the socket library
  • some way (maybe Penlight) of determining the target environment and modifying the download process/error message accordingly
Please correct me if I've missed anything.

If we can come up with a coding solution, it could also apply for libraries... although I'm hopeful that will be less of a problem when FH7 arrives. Maybe.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

@Helen, you did not look back far enough in the original thread.
It was mostly about required library modules such as pl, zip, md5, et al, which failed for various reasons including some ZIP downloads did not contain all the necessary files.
May I delete our 'ageeing' postings?
  • luacom.CreateObject("winhttp.winhttprequest.5.1") fails as winhttp is broken, so neither http nor https work
  • proxy solution suggested by Mervyn is ideal as everything would start working, so no manual downloads needed
  • http using socket library may be a solution together with a rewrite of fhloadrequire(...)
  • determining target environment is unlikely to be fruitful using Penlight app.platform() or path.is_windows
@Colin, try running the script below in the Tools > Plugins > More>> New plugin window by clicking Go.
(It relies on the pl librray module already existing in the plugins folder.)

Code: Select all

require("pl.init")
print( "OS Platform:", app.platform() )
print( "Is Windows?", path.is_windows )
On genuine Windows 10 the bottom left pane displays:

Code: Select all

OS Platform:	Windows
Is Windows?	true
Plugin has completed
I would wager you get the same in Crossover and PlayOnMac.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

Mike, the first message in the original thread was about not being able to download icon files, not libraries. And no, let's not delete stuff...

Let's see what Colin finds using your Penlight script, then decide what option to investigate next.
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by Valkrider »

Mike

In POM this is what I get. I will get the laptop out later and try Crossover but I suspect it will be the same result.

Code: Select all

OS Platform:	Windows
Is Windows?	true
Plugin has completed
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

That's a pity -- it would be useful to have a way of identifying the operating environment.
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by Valkrider »

I can now confirm that Crossover gives exactly the same result as POM.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

That is the whole point of a Windows emulator like WINE. It fools the application software into believing it is running in a Windows environment and not the host OS.

There is a catch 22 gotcha with the Socket Library Module:
  1. To perform Internet downloads we propose to use the socket library to access files via http.
  2. To use the socket library needs the loadrequire("socket") function to work.
  3. The loadrequire(...) function needs to perform an Internet file download ... so back to step 1. :o
So for the special case of loadrequire("socket") it will have to use luacom.CreateObject("winhttp.winhttprequest.5.1") which will fail in WINE and require a manual download, but at least that should be the only manual download needed.

The point about Plugin users (or even authors) not visiting the FHUG KB is not really relevant in this context.
If they use a Plugin in a genuine Windows OS they won't experience a download problem (unless something is broken).
If they are using a WINE based OS then they almost certainly would need to have followed the FHUG KB instructions at how_to:family_historian_v6_on_crossover_wine|> Family Historian V6 & Ancestral Sources V5 on Crossover, PlayOnLinux/Mac & Wine which is where the manual download processes are described.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

If they are using a WINE based OS then they almost certainly would need to have followed the FHUG KB
Fair point -- even the FaceBook user group would probably have pointed them there :D

However, as a general rule, I prefer to assume that my plugin users are totally ignorant about FHUG and the knowledgebase. I forget the figure Jane quoted to me once, but a large majority of users contacting Calico Pie for support don't know (or care) that we exist.

Ought we explore the proxy route next, before trying the socket library?
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

For the vast majority of FH & Plugin users, ignorance of the FHUG KB is unimportant, because they use a genuine Windows environment that does not suffer the WINE problems.

Yes, if we can persuade Colin or Mervyn to explore the proxy winhttp route in WINE that might yield the best outcome.
I will produce a small Plugin test script to exercise winhttp and download a file.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by ColeValleyGirl »

For the vast majority of FH & Plugin users, ignorance of the FHUG KB is unimportant
I presume you're just talking in this context? :? Otherwise, we only need the WINE pages in the new KnowledgeBase -- everything else is unimportant.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by tatewise »

Yes, just this Plugin context.

Below is a small Plugin script using luacom.CreateObject("winhttp.winhttprequest.5.1") to download my PDF icon JPG file improve_pdf_icon.jpg from FHUG KB to the Project's Public folder.
improve_pdf_icon.jpg
improve_pdf_icon.jpg (6.16 KiB) Viewed 10399 times
BTW: It also works with file Research Planner Help 2.0.zip & source http://www.fhug.org.uk/colevalleygirl/R ... lp 2.0.zip

Copy script to Tools > Plugins > More » > New… window, click Go, and check icon JPG gets downloaded.
Click X Close top right and in Save changes? prompt click No to discard or Yes to keep and choose a Plugin Name.

Code: Select all

require "lfs"
require "luacom"
local strFile = "improve_pdf_icon.jpg"
local strSource = "http://www.fhug.org.uk/pushfile.php?filename="..strFile.."&name=improve_pdf_icon&ns=/plugins/help/improve_html/"
local strTarget = fhGetContextInfo("CI_PROJECT_PUBLIC_FOLDER").."\\"..strFile
local http = luacom.CreateObject("winhttp.winhttprequest.5.1")
if lfs.attributes(strTarget,"mode") == "file" then
	local fileHandle, strError = os.remove(strTarget)	-- Delete existing file
end
http:Open("GET",strSource,false)
http:Send()
http:WaitForResponse(30)
if http.StatusText == "OK" then
	local httpSource = http.ResponseBody
	local fileTarget, strError = io.open(strTarget,"wb")
	fileTarget:write(httpSource)	-- Write source to target
	assert(fileTarget:close())
else
	error("\n Could not download chosen file from FHUG. \n "..strSource.." \n "..strTarget.." \n " )
end
The WINE HQ FAQ is at https://wiki.winehq.org/FAQ#How_do_I_co ... a_proxy.3F
The C:\Windows\System32\proxycfg.exe utility mentioned does not exist in my Windows 10 PC, probably because it has been deprecated in favour of Netsh, so presumably it only exists in a WINE system.
See https://docs.microsoft.com/en-us/previo ... v%3Dvs.85) for proxycfg.exe MSDN instructions.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Crossover/PlayonMac/Linux winhttp download problem

Post by Valkrider »

Mike

Unfortunately no luck with that plugin

Code: Select all

An error has occurred - plugin failed to complete
[string "C:\users\colin\Temp\~fh4eb4.tmp"]:19: 
 Could not download chosen file from FHUG. 
 http://www.fhug.org.uk/pushfile.php?filename=improve_pdf_icon.jpg&name=improve_pdf_icon&ns=/plugins/help/improve_html/ 
 Z:\Users\colin\Documents\Lefever ONS\Family Historian Projects\Family Historian Sample Project\Public\improve_pdf_icon.jpg 
 
No changes have been made to data records.
Post Reply