* Batch change media image extensions

[themoudie]

Thank you JP Ford for posting this query in the Media Forum.

In my previous Media Forum query Image format in FH/AS, JPG or TIF? My query included this paragraph.

Would amending the image format, within FH, require the renewal of every link between the image and data to which it is linked?

This posting would appear to answer my question and upon reading the posts, I can only say "Thank goodness, I decided to remain with a mix of JPG's and a few TIF's in my FM Projects". Trying to install "plug-ins", recognizing their shortcomings and then repairing the resulting bourach in my Project would have made me cry " 'Nough!" and walking away from the FH software.

I'll just keep to the "simple" FH/AS settings, in the hope that with software, the KISS principal remains the best.

However, I take my hat off to those who can manage these database manipulations.

Good health and happy sleuthing, BillR

[tatewise]

I've tried to fix the Search and Replace plugin to explain how the Multimedia Linked File fields and Media files filter operates and raise a warning when it is enabled. An additional warning is raised if the file extension is being replaced.

Try the prototype Search and Replace plugin Version 3.9.1 Date 16 Feb 2024.

See the updated Help & Advice page Search and Replace Extra Filters Tab with details of how the filter works.

I am sure you will let me know if that is satisfactory or not.

[Mark1834]

My two penn’orth..

To expand a little on the point I made earlier, I think Store plugins should not be able to create a situation where a file extension is inconsistent with its format. You and Helen have both claimed that it is “necessary”, but that’s a circular argument. IMO, the risk outweighs the benefit. A broken link just affects FH, not other apps that may also access the file.

Any plugin that changes a file or setting outside the “FH space” (i.e. the current Project Folder, a program system file, or an FH Registry key) should be approved only if there is a clear low-risk benefit. An example would be updating a backup file.

However, it’s CP’s ball, so we all have to play by their rules (but what are they?). If they do decide these changes are acceptable, you may want to review your warning messages.

You say what will happen, but you don’t appear to say what the potential consequences are. I would prefer to see something closer to the Windows warning that it could render the file unusable, as the typical user may not realise this.

Your warning dialog is bright purple text with no icon. We get used to your UI style of various colour text to convey additional meaning, but is purple a warning or just confirmation? What about users with impaired colour vision?

IMO, it would be better to stick closer to Windows standards - they are familiar to users and the icons immediately highlight whether it is simple confirmation or a “here be dragons” warning. If you want to customise the dialog box with additional buttons or options, it’s not difficult to create the same overall style with an IUP window. The only minor disadvantage I’ve found is that the IUP icon library hasn’t been updated so they are still old format (XP?). Interrogating the Windows API directly to get the up to date icons (11 is slightly different to 10) feels possible, but it’s above my pay grade. Hopefully somebody can point us in the right direction.

If users still go ahead with the change, I would prefer to see a prominent warning that the changes cannot be undone before they are committed, as the main FH app and some other plugins do. In the updated help, you advise reversing the options to undo the change, but what about the user who has a mix of jpg and png and wants to convert them all to png? Reversing the options doesn’t restore things to how they were - it changes every file to jpg!

[ColeValleyGirl]

I didnt argue that it was necessary to be able to change the extensions of files outside the project, but that it was necessary for plugins to be able to operate on media files outside the project.

Changing the extension is a potential problem no matter where the file is. My personal preference would be not to do it in bulk but thats just me.

[Mark1834]

It would help if CP defined what plugins can and can’t do, and under what conditions, but they don’t engage with FHUG (Simon hasn’t posted since 2022, or logged on since last summer).

Even if I asked them the specific question, I doubt I’d get a direct answer, so we’ll park the debate on fixing the root cause and focus on giving hopefully constructive feedback to help Mike fix the current problem and avoid similar issues in other plugins.

Added in edit - not directly relevant to the current issue, but a quick browse confirmed that there are well-documented methods for extracting icons via the Windows API, but I suspect they require libraries that have not been incorporated into FH (perfectly reasonably, as it’s extremely niche!).

[tatewise]

The FH Media Window has the Media > Rename Media File... command which lets the filename be changed.
(i.e. not the folder or extension.) The Help just says "Rename the media file on the hard disk."
FH makes no distinction between files inside or outside the Project folder and gives no warning about the possible impact on files shared with other products. Does that suggest plugins can do likewise?
A caveat is FH allows only one file to be changed at a time and the change can easily be reversed by Edit > Undo.

I see no distinction between files inside or outside the Project folder. They are just files on the disc.
Any of them can feasibly be shared with another product; even those inside the Project folder.

However, in a day or two I will revise the warning messages and Help page in light of the comments.

Reversing the change is possible by using the Search Scope Select Records option and choosing Media records whose Updated date-time matches the original change to limit the Search Criteria. I'll add that to the Help.
It should also be feasible to reverse the changes by using backups if necessary.

Regarding my warning dialogue, it does use an IUP window but as you say Windows icons are almost impossible.
If it mimics Windows too closely could it be mistaken for a Windows message instead of a Plugin message?
The fonts & colours can be adjusted by the user via the Set Windows Fonts button.

image.png (18.98 KiB) Viewed 602 times

[Mark1834]

I suspect we’ll never get agreement on the preferred UI style (we’re probably at the two extremes, and most users are too polite to express a preference ) or even the difference between “plugins need to be able to…” and “I want plugins to be able to…”, but how about one simple principle that all authors can sign up to -

If a plugin performs an action that cannot be reversed by Edit > Undo Plugin Updates, it goes ahead only if the user gives specific and informed consent.

Informed consent means being told not only what is about to happen, but what the potential consequences are in user language.

That excludes trivial differences such as creating or updating a configuration file - we're FH users, not attorneys!

[tatewise]

I've tried to incorporate the suggestions in the linked plugin and help below.

Try the prototype Search and Replace plugin Version 3.9.2 Date 19 Feb 2024.

See the updated Help & Advice page Search and Replace Extra Filters Tab with details of how the filter works.

[Mark1834]

I wonder if the warnings have made the help text a little complex? I had to read it several times, and I’m familiar with the process. Taking a step back, there might be a much simpler solution.

We'll park whether a plugin should be able to leave a file in a potentially unusable condition by changing the extension but not the actual format. You think it should, I think it shouldn't, and Helen is somewhere in the middle. Perhaps the question is actually "does it need to?".

Currently, the plugin prioritises maintaining the integrity of FH file links by keeping the full name of the file consistent with the FH link. However, they may not be useful links if the files have been left in an unusable condition until that is corrected by a second reformatting step outside FH.

The following protocol avoids the need to change file extensions within FH:

• If the user has requested a change that impacts only the file name and not the extension (apart from the benign changes jpg/jpeg and tif/tiff), it goes ahead after the user accepts a clear warning that file changes cannot be reversed.
• If the requested change impacts the extension, and the new version of the file exists already at the corresponding location (i.e. the user has already converted the files, as the OP did here), the change goes ahead automatically, as only FH data are affected, and it can be readily reversed with Edit > Undo plugin updates.
• If the new version of the file does not yet exist, the user gets a message telling them to either create the new files first and rerun the plugin, or proceed with the change and reformat the files later. This will be limited to updating FH links (as above), so breaking the link but with no impact outside FH, and can be reversed easily. Once they reformat the files (which will change the extension automatically), order is restored and all links are repaired.

Enforce doing it this way round doesn't take away any plugin functionality, but avoids the more contentious step altogether. It is also much easier to describe for the user in non-technical language.

[ColeValleyGirl]

We'll park whether a plugin should be able to leave a file in a potentially unusable condition by changing the extension but not the actual format. You think it should, I think it shouldn't, and Helen is somewhere in the middle. Perhaps the question is actually "does it need to?".

Helen doesn't think it should, at least not in bulk. And if there's any case to be made for it, I wonder if it ought to be broken out into a separate plugin where the pitfalls of operating on the Windows filing system can be made clearer, and it would be less likely for users to do it accidentally.

Mark's suggested protocol makes sense to me.

[Mark1834]

Dedicated plugins for that type of action is a good idea. I'm not opposed in principle to making such tools available to users who haven't got to grips with plugin authorship, just wary about what can go wrong if somebody doesn't realise the potential implications of what they are doing.

[ColeValleyGirl]

I'm conscious that splitting it out would make work for Mike, so don't suggest it lightly.

[Mark1834]

But here it is not even necessary, so it's more of a general approach to be adopted in the future.

[tatewise]

The current plugin behaviour is not far removed from Mark's three proposed scenarios.
Does the third one only apply to extension changes? If not it conflicts with the first proposal.

A further complication is that if the user clears the Confirm every item found check box to allow bulk changes to proceed all three scenarios may arise with various Media records. So presumably the plugin must pause and ask the user how to handle that case, which is then applied to all subsequent Media records.

[Mark1834]

Would it help to generalise my thinking behind the steps?

• No irreversible changes without specific and informed user consent (reversible = Edit > Undo plugin updates, not recovering from a backup).
• No changes that leave files in a potentially unstable state that depends on a subsequent step to correct, under any circumstances.
• Easy to describe for a non-technical user (explaining that oldfile.jpg is now newfile.jpg is much easier than the dangers of having a file with jpg encoding labelled as a png).
• Reasonable to implement from the current plugin with only modest changes and no loss of overall functionality, provided that steps are carried out in the correct sequence.

If you have examples where you think these principles should not apply or are contradictory, by all means put them on the table - you’re the expert on what the plugin can do!

[tatewise]

No irreversible changes without specific and informed user consent
Does each and every change need consent or can the bulk changes be consented just once?

[Mark1834]

I think it would be reasonable to allow Change All, similar to the way a search and replace box operates.

A completely different type of change probably needs a separate confirmation, but I don't think that will happen very often in a single plugin run.

[jelv]

I'd agree with most of what Mark said except this bit:

• If the new version of the file does not yet exist, the user gets a message telling them to either create the new files first and rerun the plugin, or proceed with the change and reformat the files later. This will be limited to updating FH links (as above), so breaking the link but with no impact outside FH, and can be reversed easily. Once they reformat the files (which will change the extension automatically), order is restored and all links are repaired.

I don't think plugins should ever break something deliberately, changing the extension without the file being converted is dangerous. Not all other software is as forgiving as IrfanView in allowing you to open image files with the wrong extension. Sooner or later you will get someone thinking that just changing the extension is all that is needed to change a jpg to a png or vice versa. They should be told to convert the files externally to FH and then the plugin is just repairing broken links.

[tatewise]

John, you have misunderstood that proposal the same as I did initially.
But it must be interpreted in the context of Mark's 2nd generalised thinking:
"No changes that leave files in a potentially unstable state that depends on a subsequent step to correct, under any circumstances."

So the phrase "or proceed with the change and reformat the files later" means change the Media record but not the Windows file so only FH links are changed and the link is broken but can be reversed. When the files are reformated the link gets restored.

[jelv]

OK, thanks for that explanation.

But I'm still uneasy with the idea of deliberately breaking something.

If the user converts all the files first to create copies with the appropriate extension and then batch changes the media records, at no stage is anything broken.

[Mark1834]

Yes, the original wording was probably a bit woolly, but we’re all on the same page.

Effectively, John is adding another general principle, that a plugin shouldn’t break the integrity of the current project and rely on a subsequent step to restore it.

That sounds reasonable to me, and limiting the plugin to re-establishing links to existing files forces the sequence that the OP followed - reformat the files externally, then use the plugin to fix the links. Taking away that additional element of choice may even be easier for Mike to code, and doesn’t detract from overall functionality.

[tatewise]

In light of recent discussions, I have had a radical rethink.
I've realised that sometimes the plugin may not even fulfil its own promise to avoid FH broken links.
For example:
If a Census record page holds several relevant households the user may prefer a separate Media record File link for each household that all share the same Windows file. So if the plugin Search Scope is a subset of Media Records then some of those household links may get changed and others broken!

Therefore, I propose a new set of rules that dramatically simplify the processes:

1. The plugin will never ever alter any existing Windows file under any circumstances.
   That ensures that files shared by multiple products or multiple Media records will never suffer broken links.
   Also, Edit > Undo Plugin Updates will simply revert Media record File links to their original files.
   Therefore, no user warnings are needed for those two cases.
   *
2. The plugin will only alter a Media record File link if the new value refers to a Windows file that exists.
   If necessary, the user is advised to create those Windows files and run the plugin again, but see Rule 3.
   *
3. If the File link new value does not refer to an existing Windows file the plugin may automatically create one.
   This is achieved by copying the existing file to the new file folder, path, filename, etc.
   Possibly this feature needs to be approved by the user. What do others think?
   
   If the file is an image file and the extension is being changed then the plugin may convert the file format.
   My Export Gedcom File plugin uses the Lua Image library to convert files so the same method can be used.
   Converting image files to match the changed extension may be a step too far. What do others think?
   
   For any other file extension changes the user will be advised to create those Windows files as in Rule 2.
   
   Applying this rule for file extension changes avoids warning the user about making files unusable.

Those rules will probably leave many Windows files that are not linked to Media records.
They can be resolved by running the Check for Unlinked Media (FH7) plugin and deleting redundant files.

[Mark1834]

Sounds like a good framework. Some initial thoughts -

I think I’m fairly relaxed about adding files without specific confirmation, but I can be persuaded either way.

Wary of the plugin doing conversions. What happens if a user has pdf files they want to change to images so they display in FH? Will the Lua library cope with that? An external conversion tool will probably be more flexible with more options, and if a user is not comfortable with using such a tool, they probably shouldn’t be doing automatic batch converting.

The unlinked media plugin (which will be published later this week) will be purely read only, at least in the initial version. Its greater scope comes with an enhanced risk of cock-up (e.g. accidentally deleting their entire Pictures library). I will add a short description of how users can take the output to manage their own deletions.

[tatewise]

In the Export Gedcom File plugin, I've only used the Lua IM Digital Imaging Toolkit library module to convert image files (mainly JPEG, JPG, PNG & TIFF) to image files (mainly JPEG & JPG).
See https://www.tecgraf.puc-rio.br/im/ that says it only supports BMP, GIF, JPEG, PNG, TIFF and AVI.

So document files such as DOC, DOCX, ODT, PDF, and TXT will never get converted but are in the minority.

The current Check for Unlinked Media plugin does allow deletions from the internal Project Media folder.
So that is an alternative in many cases.

[Mark1834]

I may quibble whether pdf is a purely a document format (yes, I do know what the d stands for ), as “how do I display my pdf images?” crops up here quite frequently.

However, we’re probably getting off agreeing the general principles and more into the weeds of exactly what the plugin does - that’s more down to your view of how streamlined or complex you want it to be and whether you can present it clearly to users in non-technical language…