* Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Writing and using plugins for Version 5 and above.
avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 15 May 2022 08:44

The first emulator-compatible prototype of the Backup and Restore FH Settings via Windows plugin is now available. I’ve tested it in both directions between Windows and PlayOnLinux on Mint 20.3 with a full uninstall/wipe followed by restoring a backup created by the other system. Backups were stored in a NAS Share that both systems can access directly (as \\OMV6\Shared and Z:/mnt/Shared, respectively) .

There are some limitations in the features that WINE supports, although the core Registry extract/write functions work in exactly the same way in both Windows and WINE (despite earlier incorrect claims to the contrary). I have managed to get around most of these with modifications to the code, so there are only minor differences apparent to the end user:
  • WINE does not support robocopy, so I’ve had to fall back on the much older and less flexible xcopy as an alternative in WINE. This requires an additional "discrepancy check" to check for superfluous files, as there is no "mirror" option. However, it runs virtually instantaneously even with a very slow storage medium, as we are only concerned about whether the file exists or not rather than a detailed timestamp. Changes made since backing up are always discarded on restoring, as xcopy is also missing an “ignore newer” option.
  • I haven’t found a robust way to pause a batch script for a defined period under WINE, so there are a couple of “press any key to continue” prompts when restoring.
  • “Advanced mode” is greyed out, as the Windows version is not relevant under an emulator. In principle, it should be perfectly possible to construct Linux/MacOS scripts that extract the Registry and copy the ProgramData and AppData folders directly from the host OS (e.g., by using rsync), so emulator users can exploit flexible automated backups as well. I might add that later, but let’s not run before we can walk!
  • I’ve only tested it in 32-bit prefixes, as my PlayOnLinux generates errors if I try to create 64-bit versions.
I've also fixed the 64-bit Registry bug in the Store version, and made some minor detailed improvements to the way the UI works. So it’s over to our Mac/Linux users to test please….
Mark Draper

User avatar
mjashby
Megastar
Posts: 664
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by mjashby » 15 May 2022 09:37

Mark,

I'll run some tests and let you know the results a.s.a.p.

The Crossover Build I use is based on a 32-bit Prefix, but my Linux installation uses a 64-bit Prefix and is always on the latest WINE staging build, not the 'old' versions that Ubuntu/Mint etc. tend to provide, so should add some further variety to your own tests.

Mervyn

User avatar
mjashby
Megastar
Posts: 664
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by mjashby » 15 May 2022 12:30

Mark,

I've now run the backup restore processes several times:

Windows <-> Linux <-> Crossover (Mac) <-> Windows

All processes run as you described, very speedy with only one problem. There just has to be one!

With Crossover the Backup process works with no problem, but when running the restore process, the script provides the message about closing Family Historian, it confirms the successful termination of fh.exe and then completes the restore process successfully, but the FH Interface Window simply freezes on screen. and can't be closed. It requires a 'hard closure' of FH (using Force Quit.. equivalent to using Task Manager in Windows) followed by a shutdown and restart of the Crossover to get FH running normally again (There is no damage to the app). After restarting Crossover and FH I did also check that all copying action had taken place correctly, including checking the Registry entries; and all action had been completed successfully.

My guess: Crossover has a 'pre-loader' system which registers all of its WINE apps whenever it is started, to speed up application access/startup. This may prevent some elements of a clean shutdown of FH if it sees fh.exe suddenly 'going missing' in an unexpected way. - Rather than closing the app graciously, the top border of the FH Window immediately becomes grey and although the Window can still be moved about the screen, but there is no way of directly interacting with with it. The main reason I suspect the Crossover interaction is that FH7 didn't return to operating normally until after Crossover was restarted.

It would be useful to see any other Crossover users willing to test have the same experience.

Mervyn

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 15 May 2022 12:44

Thanks Mervyn - very encouraging. What happens if you close FH manually while the first “press any key to continue” is displayed? It will then try to close something that is already closed when you continue, but hopefully it will just say that it couldn’t find the FH process and continue with the restore normally.
Mark Draper

User avatar
mjashby
Megastar
Posts: 664
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by mjashby » 15 May 2022 14:39

Mark,

Seems it may not be as much of an issue as I first experienced. More of a need to adopt a slightly different process.

Ran through this again several times and when the 'Windows' popup message appeared:
Screenshot 2022-05-15 at 14.51.32.jpg
Screenshot 2022-05-15 at 14.51.32.jpg (74.8 KiB) Viewed 789 times
1. If I chose 'yes' at this point, the the background FH Window remains greyed (and permanently frozen) as previously described but the script continues successfully. It's just necessary to 'Force Quit' the FH app using the Mac's utility and shutdown/restart Crossover to get everything working again.

2. If instead of choosing 'Yes', the FH Window is brought forward at this stage it is still active and it's possible to close the program manually. If 'Yes' is then chosen after closure the script obviously produces the error message:

Error: Could not find process "fh.exe".
Press any key to continue

Then pressing any key again allows restoration to complete successfully and FH can be restarted without having to close Crossover.

3. Choosing 'No' at any point drops back to the initial plugin screen as obviously expected.

So the real issue appears to be how to prompt Crossover Users to follow a couple of alternative steps that native Windows/Wine Users don't need to bother about. You obviously didn't experience this with PlayOnLinux so it may be only apparent when using Crossover.

Hope that helps.

Mervyn

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 15 May 2022 17:18

Indeed it does, thank you. I think there are two simple refinements that are worth making to improve the user experience under WINE. Firstly, make the auto close of FH selectable, with an additional option box that only WINE users will see (default on, as it is ok under both PlayOnLinux and Windows). Secondly, it is always possible that a future WINE upgrade will support robocopy, so I can add a silent check to see if that is available and only default to the more limited xcopy if not.

It's not surprising that we are seeing more variability than is normal, as virtually all other plugins only interact with FH. Here, we are getting up close and personal with the underlying OS.
Mark Draper

User avatar
mjashby
Megastar
Posts: 664
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by mjashby » 15 May 2022 17:33

Mark,

O.K. I'll watch for any updates and retest as necessary.

Well done.

Mervyn

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 18 May 2022 08:35

I've changed my mind on both those updates :).

Rather than going down the road of "option bloat" and introducing an option that may confuse more people than it help, I've removed the automatic FH close for all WINE users. You now get a clear prompt about when to close the program manually, which more or less mirrors what Mervyn was doing anyway.

I will also delay any check for robocopy under WINE until it is actually introduced, rather than trying to second guess how it will be implemented.

There is also an important enhancement to the options with this version, as I have reinstated the ability to retain changes to settings made since the backup was created. I had misinterpreted how one of the xcopy switches worked. The downside is that updates to backups now require a full recopying of all files, but it is very quick for anything other than the slowest storage medium, and as in Windows, FH remains fully useable while the copying is in progress.

I have also tweaked the Restore function so the plugin never overwrites its own settings or any of the temporary working files. That is a lot more convenient where the backup and restore PCs have different folder structures and backup locations.

I did encounter one problem testing its ability to restore backups from the old plugin. Unfortunately, the old plugin does not run correctly in my "out of the box" PlayOnLinux setup. It loads ok, but generates a WINE error as soon as it starts the backup.
Screenshot from 2022-05-18 09-05-31.png
Screenshot from 2022-05-18 09-05-31.png (76.7 KiB) Viewed 708 times
Does it write to 64-bit Registry keys explicitly? That can cause the same symptoms, and was why my recent simple WINE detector was failing for both Mervyn and Colin. Any other uses having issues with the plugin in 32-bit WINE prefixes?

I haven't tried the old plugin in FH6/WINE, as that requested I download and install md5, which immediately generated an error.

The new plugin in FH7/WINE successfully restores a backup created in FH7/Windows with the old plugin.

Over to users - any issues, please...?
Last edited by tatewise on 23 May 2022 09:13, edited 1 time in total.
Reason: Attachment deleted as later version is in the Plugin Store.
Mark Draper

User avatar
mjashby
Megastar
Posts: 664
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by mjashby » 18 May 2022 11:03

Mark,

Have tested on both:

- Crossover 21.2, which i believe still uses Wine 7.0 'Stable'; and
- Linux (Debian), using Wine 7.8 (Staging) which contains the very latest developments/bugfixes under testing. - Which seems to have the added 'advantage' of being able to use FH's internal Browser Window.

Both export and import worked with no problems; apart from user introduced PEBCAKs (Problem Exists Between Chair and Keyboard) :D , by not following the steps in the precise sequence required for shutting down FH. As before, the Linux/Wine installation import also worked successfully when allowing the script to auto-shutdown FH.

All Good for me!

I did explore further to see if I could identify what might be causing the Crossover issue and noticed that under Linux there is a warning message when FH is shutdown that fh.exe has been 'forcibly terminated'. As Crossover doesn't create standalone Wine.apps, but applications which remain dependent on Crossover to function, it appears to 'object' to its control of applications being 'usurped' by a 'third party' and so locks the app screen, which then requires a Crossover restart re-establish control. Other 'front-end' Wine Management applications might be more forgiving.

Mervyn

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 18 May 2022 12:12

Thanks Mervyn, appreciated. It sounds like this version is probably good to go, but I'll leave it on here for a few days before submitting to the Store in case anybody else wants to test it (there are detailed improvements and bug fixes under Windows as well, so all users will benefit from the upgrade when it is submitted).

It does sound like a CrossOver problem that is affecting the autoclose. I'll play with raw WINE on other distributions sometime later, but that's more for general interest rather than any burning need to update the plugin. This version appears fully functional under both Windows and WINE, so if it ain't broke, I'll resist the temptation to fix it... ;)
Mark Draper

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 19 May 2022 08:43

I was doing some housekeeping to clear up test files from my plugin development, and tried a simple test of stepping through the old plugin to see where the WINE error was occurring.

Here's the answer. The file referred to is also attached. As far as I can see, it's just a copy of the plugin script. If I close the window, WINE crashes. Any help?
Screenshot from 2022-05-19 09-34-58.png
Screenshot from 2022-05-19 09-34-58.png (68.3 KiB) Viewed 623 times
Attachments
~fh7819.tmp.zip
(66.98 KiB) Downloaded 8 times
Mark Draper

User avatar
tatewise
Megastar
Posts: 24400
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by tatewise » 19 May 2022 15:21

Yes, that is a problem I run into with my Plugins that use my FSO based version of the DirTree(...) function.
It usually only arises when run in debug mode and seems to be an internal Lua bug of some sort.
Where possible I now use the FSO:GetFolder(...) function and loop on its resultant table instead of using the Lua coroutine.yield(...) function, but it makes the script less transparent and can be much slower.

This problem does NOT arise in FH v6.2 Lua 5.1 with exactly the same script.

I reported the problem to CP who said "Thank you for your feedback."
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 19 May 2022 18:44

What other reply were you expecting? If it is a Lua bug, CP can't be expected to fix it. Their priority will be to improve the provided tools such as fhFileUtils in the current version of FH, rendering such diy alternatives unnecessary.

For me, it illustrates my design philosophy perfectly - a mass of complex and buggy code, with a huge performance overhead, just to make Lua do what Windows does in a couple of lines. It could also have more general implications for emulator users, as it makes plugins less compatible with WINE.
Mark Draper

User avatar
tatewise
Megastar
Posts: 24400
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by tatewise » 19 May 2022 19:46

Sorry, Mark, you misunderstand.
For many years the Code Snippet Directory Tree has worked fine based on the lfs library but only supports ANSI file paths.
My more complex plugins all use the same library of Code Snippets that I know to work for me.
The fhFileUtils does not include a substitute for that Directory Tree Code Snippet to support UTF-8 file paths.
So I substituted FSO functions instead of lfs functions to produce a compatible Code Snippet such that my existing plugins did not need to change the calling script, but would support UTF-8 file paths.

When it failed I could not be entirely sure whether it was an Lua fault, or an FH implementation bug, or what.
So I reported it to CP in case they could repeat the problem and maybe find a solution.
From their reply, that is not likely to happen. So I am having to design a workaround for each of my plugins.
Also, I cannot use fhFileUtils and maintain backward compatibility with FH V5 and V6.
The problem does not only affect my Backup and Restore FH Settings plugin but several others.

I was simply responding to your posting about the error and explaining my experience.
Perhaps I should have just kept quiet.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 19 May 2022 21:57

No right or wrong answer on this one. My view is that adding all this complexity to support a small and dwindling market can end up creating problems for everyone else. You are loyal to your legacy FH5/6 user base and put in a huge amount of effort to provide plugin upgrades for those who have made a conscious decision not to embrace FH7.

It’s a definite plus for FH that it is flexible enough to accommodate both approaches, and users can adopt whichever they feel more comfortable with.
Mark Draper

User avatar
tatewise
Megastar
Posts: 24400
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by tatewise » 20 May 2022 10:45

Mark, you are still missing the point. Regardless of any backward support for FH V5 and V6, the problem afflicts FH V7 because fhFileUtils does not provide a substitute for the Code Snippet Directory Tree. So even if my plugins abandoned FH V5 and V6 the problem would still exist.

It seems to be a problem with Lua 5.3 module 6.2 – Coroutine Manipulation that is used by the Code Snippet Directory Tree but could also be used elsewhere with similar dire consequences.

I have not found a suitable general solution using fhFileUtils to traverse folders and files efficiently without having to write specific DIY scripts for each case depending on what needs to be processed.

I applaud your new plugin which benefits from advances in FH V7 that force Diagram Icons into the ProgramData folder, developments in WINE that appear to offer better Registry access, and other neat ideas, which will allow me to retire my plugin sooner rather than later. FH V5 and V6 users could continue using the latest version without problems, whilst FH V7 users would use your plugin.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 20 May 2022 13:58

No, I am aware that fhFileUtils has significant gaps, and said so above. The difference is that you are keen to use that missing functionality now, so have developed your own method for doing so. I know that CP are fully aware of the fhFileUtils issues, and am confident (albeit without any direct evidence) they will be addressed for FH7.

I recognise that you are probably in a unique position, as I doubt that any other author will want/need to retrofit this into FH5/6, but it does mean that you accept sole responsibility for debugging and maintenance, as has been discussed before. Intuitively, that doesn’t feel quite right for an open-source product, but how you spend your time is nobody’s business but yours (and possibly Mrs Tate’s ;)).

In terms of who could use what today, the only folk who don’t have a choice are FH5 users (old only), and possibly FH7 emulator users (new only) for the WINE compatibility issue discussed above. I deliberately restricted file management to lfs and the io library in order to not break FH6, as it doesn’t need UTF-compatibility. It adds a few lines of extra code, but it’s simple code that any other author could understand and adapt very easily.

The only reason that FH5 is excluded in the new plugin is that I use FH6+ functions to check for ASCII/ANSI compliance. Reading that function again, it would be simple (and a worthwhile simplification in its own right) to just check byte values directly (all between 32 and 127, inclusively). I’ll do that before I submit the emulator upgrade to the Store next week. That would give the new plugin universal application.
Mark Draper

User avatar
Jane
Site Admin
Posts: 8340
Joined: 01 Nov 2002 15:00
Family Historian: V7
Location: Somerset, England
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Jane » 20 May 2022 16:31

tatewise wrote:
20 May 2022 10:45
fhFileUtils does not provide a substitute for the Code Snippet Directory Tree.
I might be missing something important, but does getFolderContents() not offer similar options

Code: Select all

local sProjectFile =  fhGetContextInfo('CI_PROJECT_FILE')
local sProjectPath =  fhf.getParent(sProjectFile) 
local sPath
print(sProjectFile,sProjectPath, #sProjectPath)

local fileList =  fhf.getFolderContents(sProjectPath,true)
for _,filedata in ipairs(fileList) do
	sPath = filedata.path
	if #sPath > 200 then
		print(#sPath,sPath)
   end
end
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 20 May 2022 17:20

There is an even easier way that works in all versions of FH

Code: Select all

local cmd = 'dir "' .. fhGetContextInfo('CI_APP_DATA_FOLDER') .. '" /s/b'
local R = io.popen(cmd)
local Files = R:read('*a')
R:close()

print(Files)
However, the limitation of both these methods is that they only provide the file name/path. What's missing is being able to extract other attributes, such as creation and modification time stamps (ideally in a Lua-compatible "seconds since start of epoch" format). IMO, it's a major omission from a UTF-compatible file and folder management library.
Mark Draper

User avatar
Jane
Site Admin
Posts: 8340
Joined: 01 Nov 2002 15:00
Family Historian: V7
Location: Somerset, England
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Jane » 20 May 2022 18:23

From fhfilutils help
getFolderContents (sPath, bRecurse)
Return table of all files and folders in a specified folder. For each item the type,path,shortname, shortpath, created date,modified date,size and attributes are returned in a table
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 20 May 2022 18:45

Yes, but the created and modified dates are returned as text strings that depend on the local Windows settings. How do we convert those to locale-independent values that can be used for calculation (is file a older or newer than file b?, how long ago was file a modified?, etc)...?
Mark Draper

User avatar
tatewise
Megastar
Posts: 24400
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by tatewise » 20 May 2022 19:24

Yes, Mark is correct. The getFolderContents (...) function is far from a substitute for the Code Snippet Directory Tree.
luacom.DateFormat = "table" must be used to get a locale independent date-time text string and then os.time(...) used to convert it to the epoch seconds since 1/1/1970.
That requires major changes to existing plugins to replace the DirTree(...) calling script.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 3688
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by ColeValleyGirl » 20 May 2022 19:29

We have been over this ground repeatedly. I remember pointing out the luacom. Date format option to Mike as a way forward for his own use

viewtopic.php?f=42&t=20497&p=123571#p123571

avatar
Mark1834
Megastar
Posts: 1454
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Mark1834 » 20 May 2022 19:36

Indeed - it would be put to bed for good by a public acknowledgement from CP that they recognise the issue and plan to fix it, but radio silence from them...
Mark Draper

User avatar
Jane
Site Admin
Posts: 8340
Joined: 01 Nov 2002 15:00
Family Historian: V7
Location: Somerset, England
Contact:

Re: Mac/Linux-compatible version of Backup & Restore FH Settings via Windows plugin

Post by Jane » 20 May 2022 20:33

As you will see from the fhfileutils, Helen and I created and submitted to Calico the original code for this. When time allows I will try and work out an enhancement to submit for consideration based on Helens previous suggestion, the limitation is you can only work with the APIs available from Windows. It is very disappointing that you both seem to think they are no good. You could always simply work with the windows APIs directly rather than using the modules.
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

Post Reply