* Check for Possible Duplicate Media error

[ColinMc]

I ran this plugin this morning, and it crashed with the following error

plug 1.jpg (27.81 KiB) Viewed 1405 times

Just in case there was a memory issue, I re-booted and ran it again, and still got a crash.

As I have 32Gb of RAM, this did surprise me. On checking (after the crash) Family Historian is showing 18% memory usage, and when I checked while the plugin was running, it never rose above 21% Memory usage before crashing.

Is there any log that can point to where it is falling over?

Colin

[tatewise]

Are you prepared to perform a little self-diagnosis?

In Tools > Plugins... click the More >> button to reveal the side panel.
Scroll down and select the Check for Possible Duplicate Media plugin, then click Edit... in the side panel.
In the editor window scroll down to line 230 to the function LoadFromFile to reveal line 233 stated in the error message.
Insert the print(strFileName) statement as shown below, which reports each file whose contents is being read.

Code: Select all

function LoadFromFile(strFileName)
    -- Load string from file --
print(strFileName)
    local fileHandle = OpenFile(strFileName,"rb")
    local strString = fileHandle:read("*all")
    assert(fileHandle:close())
    return strString
end -- function LoadFromFile

Now click the Go button top left to run the plugin in debug mode.
In the editor window, the bottom left pane will list each file inspected until the memory crash.
The last file listed is probably the cause due to something unusual in its contents.

If that does not correctly identify a problem file, then some other debug tactics may be needed.
There is something about a particular file in your Media folder that is causing the upset so it will be difficult to discover what that is without a copy of all your files.

[ColinMc]

Thanks for your help, and the clear instructions.

It fell over at an MP4 file it came across.

plug 2.jpg (61.21 KiB) Viewed 1368 times

I only have 3 video files in my project, (all MP4), and on checking the list of files processed, that files was the first MP4 the plugin encountered. File size was 23.9Mb, so on its own, I would have thought that that should not have caused a memory issue.

PS I suspect it may not be relevant, but just in case, the location for my Media files on my C drive is a virtual folder created by a Directory junction.

[tatewise]

I don't think the Directory Junction is a problem, especially as all the other Media files are processed OK.

To confirm that the MP4 files are the problem, please move the first one to some other folder temporarily.
Does the Plugin then crash on the next MP4 file?
If they are all moved to another folder does the Plugin complete OK?
Remember to move them all back.

As a workaround, change that function LoadFromFile as follows and File > Save the plugin script.
i.e. Insert the new statement if strFileName:match("mp4$") then return strFileName end

Code: Select all

function LoadFromFile(strFileName)
    -- Load string from file --
    if strFileName:match("mp4$") then return strFileName end
    local fileHandle = OpenFile(strFileName,"rb")
    local string = fileHandle:read("*all")
    assert(fileHandle:close())
    return strString
end -- function LoadFromFile

That pretends that the full file path & name is its contents, so any duplicate MP4 files will not get detected.

Hopefully, Jane the plugin author, will see this thread and investigate a solution.

[ColinMc]

Ran the plugin again as suggested, and as you suspected, it failed on the next MP4, then finally on a third run, on the last one.

Adding the files back in caused a fail again.

Interestingly, when all three of the MP4's were removed, the plugin ran to completion, and reported (amongst a few other semi-expected duplicate jpg listings), that the three MP4 files were missing.

I'll add your work-around this evening when I get a chance.

As ever, thanks for your help and expertise.

[Jane]

The plugin loads the files into memory to calculate the MD5 to check for duplicates which is why it is failing on those large MP4 files, I will consider simply skipping the MD5 check on files which are larger than normal images.

[ColinMc]

Many thanks for investigating. Still seems odd that a relatively small file at 23Mb would cause a memory error when I'm only running at approx 20% of 32Gb.

[Jane]

Remember LUA is running in a limited environment under FH it does not have access to all the memory on your PC.

If you want to avoid the problem just add the code Mike suggested.

[ColinMc]

Remember LUA is running in a limited environment under FH it does not have access to all the memory on your PC.

I have no knowledge about LUA and therefore no idea what LUA is running in, so would never have been aware of that, so thanks for the explanation.

I'll add the code this evening as suggested.

Thanks again, problem solved.

[johnmorrisoniom]

I had been getting the same error, but been to busy to anything about it.
Moved the offending MP4 file out of the project (159 Mb) and plugin ran correctly.

Sorted out the duplicates reported na d moved the MP4 back into place and add the line of code supplied by Mike, all now OK.

Before the plugin was updated for FH7 it used to cache results so that it would run quicker next time it was run. Is there anyway to add that feature back in. I have over 42,000 media records so the plugin can take 30 minutes to run.

[ColeValleyGirl]

Will the new feature in 7.0.17 that prevents duplicate media being created make it unnecessary to run it again?

[Mark1834]

Does the new in-built feature check file contents, or just prevent creating a new Media Record if there is another record containing the specified file name and path?

If Jane is updating the plugin, is it worth including an option to check just for that scenario - the same file name and path in multiple media records? That would run extremely quickly, so solve John's problem as well.

[johnmorrisoniom]

Most of my problems with duplicate media are self inflicted.. Usually by attaching the wrong image to a source and renaming it to match the source.
In the last batch of duplicates I had imported a census image as a marriage, imported Marriage image as a census, imported an 1891 census as a 1911 image. I don't know how CP's new features will help.

All these were probably done late at night when tired and being a bit cross eyed.

[ColeValleyGirl]

Does the new in-built feature check file contents, or just prevent creating a new Media Record if there is another record containing the specified file name and path?

It doesn't check file contents, as far as I can tell. And of course it won't help with images added via AS.

[tatewise]

See Check for Possible Duplicated Media (19256) that has an attached plugin.
It explains why the cache was lost due to the loss of Lua SQL and adds a data cache so that it would run quicker.

I had hoped that its features would get incorporated into the Plugin Sore version by Jane.