* Finding and Deleting duplicated weblinks with Search and Replace?

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
Post Reply
avatar
MFriend
Famous
Posts: 111
Joined: 30 Jan 2021 07:43
Family Historian: V7

Finding and Deleting duplicated weblinks with Search and Replace?

Post by MFriend »

Hi Folks:
I've been working on fixing up my wifes data on her side of the family. One issue I have is after importing from RM 8, I have weblinks that are duplicated on almost all citations (which is about 56,000). Here is what I'm talking about:

Image

I figure there is a way using the search and replace plugin to delete the duplicate link. I know how limit the search and replace to just that part of the citation, I just don't know how to configure Search and replace to select and delete the duplicate link (assuming that is possible which it seems almost anything is possible with FH 7 or FH7 plugins).

Matthew

(P.S. this is not a problem caused by the import process. The data came from FTM 2019 and was exported to RM 8. Somehow along the way the weblinks in FTM 2019 were saved as both a weblink and a note in RM 8. FH7 when importing seems to combine the two resulting in a duplicate in that section)
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Finding and Deleting duplicated weblinks with Search and Replace?

Post by tatewise »

The Search and Replace plugin can be quite clever but detecting duplicated data is beyond its capabilities.
There are several published plugins that detect duplicates of various entities but none will find your duplicate web links.

A custom plugin will need to be written to perform the detect and delete process.
Are the duplicates always in a Web Links citation-specific metafield and are such metafields always of Type URL?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
MFriend
Famous
Posts: 111
Joined: 30 Jan 2021 07:43
Family Historian: V7

Re: Finding and Deleting duplicated weblinks with Search and Replace?

Post by MFriend »

Hi Mike:
Yes, they are always Weblinks and always type URL :)

Matthew
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Finding and Deleting duplicated weblinks with Search and Replace?

Post by tatewise »

On looking at this more closely, I think I have misunderstood the Web Links field details.
I don't think they are Citation-specific Source Template metafields at all.

I suspect you have customized the Citation Window so that it displays Web Links: on the Citation tab.
Furthermore, I suspect that is actually the local Citation Note field with Data Reference: SOUR.NOTE2
Please confirm, preferably with a screenshot of the Web Links: customization Edit Item window.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
MFriend
Famous
Posts: 111
Joined: 30 Jan 2021 07:43
Family Historian: V7

Re: Finding and Deleting duplicated weblinks with Search and Replace?

Post by MFriend »

Hi Mike:
I hadn't noticed that section had been changed also. In RM8 I only used the default generic templates to avoid issues when moving to online trees, etc. with sources and citations.

FH 7 during the direct import I guess makes some changes from the default.

Here is a pic of that field:
Image

thank you,
Matthew

Unrelated I think, but even though I didn't use any special templates in RM8, it still imported the citation specific "Where within sourc" into the <Template Fields> section where is says ".Page: " I think the direct import though did do a good job of ensuring all the data made it into the new FH7 database without losing data.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Finding and Deleting duplicated weblinks with Search and Replace?

Post by tatewise »

It seems that the RM8 import process has customized the Citation Window as follows:
Source Record
Title: no change
Template: no change
• Footnote: '<Template Fields>' definition from Source Template
• Short Footnote: ditto
• Bibliography: ditto
Source Text: 'Text from Source' definition with a new label (SOUR.TEXT generic field)
Citation-specific Details
• Page: 'Citation Details: <Template Fields>' definition from Source Template
Citation: 'Citation Details: Where Within Source' definition with a new label (SOUR.TEXT generic field)
Citation Text: 'Citation Details: Text from Source' definition with a new label (SOUR.DATA.TEXT generic field)
Web Links: 'Citation Details: Note' definition with a new label (SOUR.NOTE2 generic field)

I have written a custom plugin to handle duplicate Web Links that offers to list or delete the duplicate.
Try the attached Find Duplicate Web Links plugin Version 0.1 Date 31 May 2022.
It would be wise to create a backup before running the plugin.
Afterwards, the Edit > Undo Plugin Updates command will reverse its effect but only before closing FH.
Last edited by tatewise on 01 Feb 2024 18:10, edited 1 time in total.
Reason: Attachment deleted - contact Mike Tate if needed
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
MFriend
Famous
Posts: 111
Joined: 30 Jan 2021 07:43
Family Historian: V7

Re: Finding and Deleting duplicated weblinks with Search and Replace?

Post by MFriend »

Thank you Mike, that worked perfect.

You saved me having to manually delete thousands of dupes :)

Matthew
Post Reply