Family Historian User Group

Helping Family Historian users since 2002
https://fhug.org.uk/forum/

Data References ~ Evernote ~ Research Notes

https://fhug.org.uk/forum/viewtopic.php?f=42&t=18524

Page 2 of 2

Re: Data References ~ Evernote ~ Research Notes

Posted: 01 Jan 2021 18:06
by gsward
Mike, thanks for your pointers on the extra <div> </div> tags and more careful formatting of my notes.

I am now at a point where I have an Evernote file (enex) created, visually it appears to be correctly formatted but fails to actually import. As I mentioned earlier there are some issues with Evernote itself at the moment to the point they have released their legacy version 6 alongside the latest version 10.

Now version 6 gives an error message "missing_note_content" but even after a lot of searching I have been unable to establish exactly what causes this error. Version 10 does not give an error message at all, but it creates a new notebook but no note within! This is even more strange as Microsoft OneNote imports the enex file perfectly and creates a correctly formatted page.

So it's with me to try and get to the bottom of the Evernote issue but it may not be quick as Evernote's forums and support are awash with disgruntled users.

If there is any other FH user who also uses Evernote and wants to take a look, let us know.

Thanks.

Re: Data References ~ Evernote ~ Research Notes

Posted: 01 Jan 2021 18:26
by tatewise
So, essentially the plugin works correctly as a prototype but will need refining to cope with whatever Evernote eventually requires plus the other cases we identified such as format codes spanning multiple lines.
In another thread, my Export Gedcom File conversion of rich text FTF has been discussed and needed some correction, which will need to feed into this Evernote plugin and maybe a Knowledge Base plugin code snippet.

In the meantime, Graham, can you attach your latest version of the plugin here for others to take a look.

Re: Data References ~ Evernote ~ Research Notes

Posted: 01 Jan 2021 20:08
by gsward
Evernote_to_do 0.9.2.fh_lua
(26.72 KiB) Downloaded 97 times
Here is a prototype version.

Please read the discussion above and be aware that Evernote currently has its own problems whilst they are running two versions. This plugin will only work with version 6.x, 'Evernote Legacy'.

The significant change in this version to previous versions of the plugin is the ability to handle Research Notes (in Rich Text Format) linked to an individual.

Graham

Re: Data References ~ Evernote ~ Research Notes

Posted: 03 Jan 2021 19:29
by gsward
Mike,

I am still working on testing what we have so far and trying to understand the quirks of how Evernote handles urls.

One thing I have sorted, I have discovered that of <col> is used in a table Evernote doesn't require <colgropup> to be there but it does expect a closing tag </col> which is not the normal html practice. anyway I have sorted that.

I have also discovered I need to change the <th> to <td> in the tables section. It works, but it is not ideal as it is because it centres and puts the text in bold by default.

Now my grasp of pattern matching is not up to editing this section fully yet, the first line works OK but in the "Alignment" block we need both the opening and closing <th> </th> to change to <td> </td>. Could you advise how it needs to change?

Code: Select all

strAny, intAny = strAny:gsub('<row>(.-)</row>\r',function(strRow) return '<tr><td>'..(strRow:gsub('|','</td><td>'))..'</td></tr>' end) 
intSum = intSum + intAny

-- Alignment <align="r"> <align="c"> <align="j">
strAny, intAny = strAny:gsub('([^\r])<align="r">(.-)</th>','%1<div style="text-align:right;">%2</div></th>') 
intSum = intSum + intAny
strAny, intAny = strAny:gsub('([^\r])<align="c">(.-)</th>','%1<div style="text-align:center;">%2</div></th>') 
intSum = intSum + intAny
strAny, intAny = strAny:gsub('([^\r])<align="j">(.-)</th>','%1<div style="text-align:justify;">%2</div></th>') 
intSum = intSum + intAny
strAny, intAny = strAny:gsub('<align="r">(.-)\r','<div style="text-align:right;">%1</div>\r') 
intSum = intSum + intAny
strAny, intAny = strAny:gsub('<align="c">(.-)\r','<div style="text-align:center;">%1</div>\r') 
intSum = intSum + intAny
strAny, intAny = strAny:gsub('<align="j">(.-)\r','<div style="text-align:justify;">%1</div>\r') 
intSum = intSum + intAny
Thanks.

Re: Data References ~ Evernote ~ Research Notes

Posted: 03 Jan 2021 20:24
by tatewise
You are largely correct about the use of <th> & </th> being <td> & </td>
The former should only be used for heading rows or columns so the latter is better.

In that code block just replace </th> with </td> as you did in the first line.

Other changes have arisen as part of Export Gedcom File HTML testing.

Firstly, every <div & </div> should be <span & </span>

The code for Colour and Highlight need to become:

Code: Select all

			-- Text Colour <clr="FF0000"> <clr="0000FF"> </clr> not in matched pairs
			strLine 			= strLine:gsub('<clr="(.-)">', '</clr><clr="%1">')
			strLine, intAny	= strLine:gsub('<clr="(.-)">(.-)</clr>', '<span style="color:#%1;">%2</span>')
			intSum = intSum + intAny
			strLine 			= strLine:gsub('</clr>', '')

			-- Highlight Colour <hl="0000FF"> <hl="FF0000"> </hl> not in matched pairs
			strLine 			= strLine:gsub('<hl="(.-)">', '</hl><hl="%1">')
			strLine, intAny	= strLine:gsub('<hl="(.-)">(.-)</hl>', '<span style="background-color:#%1;">%2</span>')
			intSum = intSum + intAny
			strLine 			= strLine:gsub('</hl>', '')
and for Font Size and Style:

Code: Select all

			-- Font Size <fs="+4"> <fs="-4"> </fs> not in matched pairs and need size adjusting by +10pt
			strLine 			= strLine:gsub('<fs="%+(%d+)">', function(strSize) return '<fs="'..string.format("%d",10+strSize)..'">' end )
			strLine 			= strLine:gsub('<fs="%-(%d+)">', function(strSize) return '<fs="'..string.format("%d",10-strSize)..'">' end )
			strLine 			= strLine:gsub('<fs="(.-)">', '</fs><fs="%1">')
			strLine, intAny	= strLine:gsub('<fs="(.-)">(.-)</fs>', '<span style="font-size:%1pt;">%2</span>')
			intSum = intSum + intAny
			strLine 			= strLine:gsub('</fs>', '')

			-- Font Style <font="Calibri","swiss",0> <font="Calibri"> </font> not in matched pairs and need all " string quotes and trailing digits removing
			strLine 			= strLine:gsub('<font=".->', function(strFont) return ( strFont:gsub('"',''):gsub(',+%d->$','>') ) end )
			strLine 			= strLine:gsub('<font=(.-)>', '</font><font=%1>')
			strLine, intAny	= strLine:gsub('<font=(.-)>(.-)</font>', '<span style="font-family:%1;">%2</span>')
			intSum = intSum + intAny
			strLine 			= strLine:gsub('</font>', '')

Re: Data References ~ Evernote ~ Research Notes

Posted: 07 Jan 2021 19:45
by gsward
Mike,

I have got back to some testing and trying to find how imports into Evernote fail. Usually when they do fail, Evernote puts its failed version of the enex with an error message in a temporary folder, but not always!

URLs

One of the ones that was puzzling me was things go wrong when the Research Note includes a url. Anyway, I have fixed that one.

Code: Select all

strLine, intAny = strLine:gsub('<web="([^"]+)"?,?"?([^"]-)">','<a href=%1>%2</a>')
needs to be

Code: Select all

strLine, intAny = strLine:gsub('<web="([^"]+)"?,?"?([^"]-)">','<a href="%1">%2</a>')
<th> and <td>

We have touched on this one before. We now have:

Code: Select all

strLine, intAny = strLine:gsub('<row>(.-)</row>$',function(strRow) return '<tr><td>'..(strRow:gsub('|','</td><td>'))..'</td></tr>' )
In the section that follows which addresses Alignment I am still not clear how to adapt the code to change both the opening and closing <th> </th> to <td> </td> as we are only covering the closing </th>.

Code: Select all

-- Alignment <align="r"> <align="c"> <align="j">
strLine, intAny = strLine:gsub('(..-)<align="r">(.-)</th>','%1<div style="text-align:right;">%2</div></th>') 
intSum = intSum + intAny
strLine, intAny = strLine:gsub('(..-)<align="c">(.-)</th>','%1<div style="text-align:center;">%2</div></th>') 
intSum = intSum + intAny
strLine, intAny = strLine:gsub('(..-)<align="j">(.-)</th>','%1<div style="text-align:justify;">%2</div></th>') 
intSum = intSum + intAny
strLine, intAny = strLine:gsub('<align="r">(.-)$','<div style="text-align:right;">%1</div>') 
intSum = intSum + intAny
strLine, intAny = strLine:gsub('<align="c">(.-)$','<div style="text-align:center;">%1</div>') 
intSum = intSum + intAny
strLine, intAny = strLine:gsub('<align="j">(.-)$','<div style="text-align:justify;">%1</div>') 
intSum = intSum + intAny
Tables

For a successful import I have found that tables don't need <div> and </div> either on each line or across the block.

Now the <div> is put in here at the end of the section.

Code: Select all

table.insert(arrLine,"<div>"..strLine.."</div>")
If I take the <div> </div> out the table works but the other lines that needed <div> all fail!

So as an example I don't want this:

Code: Select all

<div>Title: Research <b>Note</b> with urls and table</div>
<div>Type: <u>Tasks</u> </div>
<div>Repository: </div>
<div>Date: </div>
<div>Description: <i>Grave</i> </div>
<div>Status: <s>open</s> </div>
<<div><br/></div>
<div><br/></div>
<div><table><col style="width:786px;"></col><col style="width:171px;"></col><col style="width:126px;"></col></div>
<div><tr><td> Task Description </td><td> Status </td><td> Date Completed </td></tr></div>
<div><tr><td> Record: "Elizabeth MAYES"  </td><td>  </td><td>  </td></tr></div>
<div><tr><td> Wife's grave? </td><td>  </td><td>  </td></tr></div>
<div><tr><td>  </td><td>  </td><td>  </td></tr></div>
<div><tr><td>  </td><td>  </td><td>  </td></tr></div>
<div><tr><td>  </td><td>  </td><td>  </td></tr></div>
<div><tr><td>  </td><td>  </td><td>  </td></tr></div>
<div></table></div>
<div><br/></div>
but this:

Code: Select all

div>Title: Research <b>Note</b> with urls and table</div>
<div>Type: <u>Tasks</u> </div>
<div>Repository: </div>
<div>Date: </div>
<div>Description: <i>Grave</i> </div>
<div>Status: <s>open</s> </div>
<<div><br/></div>
<div><br/></div>
<table><col style="width:786px;"></col><col style="width:171px;"></col><col style="width:126px;"></col>
<tr><td> Task Description </td><td> Status </td><td> Date Completed </td></tr>
<tr><td> Record: "Elizabeth MAYES"  </td><td>  </td><td>  </td></tr>
<tr><td> Wife's grave? </td><td>  </td><td>  </td></tr>
<tr><td>  </td><td>  </td><td>  </td></tr>
<tr><td>  </td><td>  </td><td>  </td></tr>
<tr><td>  </td><td>  </td><td>  </td></tr>
<tr><td>  </td><td>  </td><td>  </td></tr>
</table>
<div><br/></div>
Could you kindly assist, please?

Graham

Re: Data References ~ Evernote ~ Research Notes

Posted: 07 Jan 2021 20:24
by tatewise
URL
That correction I had also spotted in my plugin.

Alignment
For that only </th> needs changing to </td>
It is just looking for <align="?"> prior to cell end </td> or line end $ there is no <td> involved

BTW: Every div in that section should be span

Tables
In stead of table.insert(arrLine,"<div>"..strLine.."</div>") use this to remove div from table lines:

Code: Select all

	strLine = "<div>"..strLine.."</div>"
	strLine = strLine:gsub('^<div><table>(.*)</col></div>$','<table>%1</col>')
	strLine = strLine:gsub('^<div><tr>(.*)</tr></div>$','<tr>%1</tr>')
	strLine = strLine:gsub('^<div></table></div>$','</table>')
	table.insert(arrLine,strLine)

Re: Data References ~ Evernote ~ Research Notes

Posted: 07 Jan 2021 22:00
by gsward
Thank you. That all works in my test file now. When I have confirmed a few other scenarios I'll post the updated plugin.

Graham

Re: Data References ~ Evernote ~ Research Notes

Posted: 08 Jan 2021 10:23
by gsward
As I suspected, we still have an issue with the Alignment section.

If I, say, set align:right in the table the import fails.

It generates a line:

Code: Select all

<div><tr><td> Task Description </td><td> <span style="text-align:right;">Status </td><td> Date Completed </td></tr></span></div>
I thought I could fix this by adding a new line to the section in your previous reply

Code: Select all

strLine = strLine:gsub('^<div><tr>(.*)</span></div>$','<tr>%1</span>')
but that generates a line that fails:

Code: Select all

<tr><td> Task Description </td><td> <span style="text-align:right;">Status </td><td> Date Completed </td></tr></span>
because the </span> is at the end of the line.
If I manually reposition </span> to this:

Code: Select all

<tr><td> Task Description </td><td> <span style="text-align:right;">Status </span></td><td> Date Completed </td></tr>
It imports without error but the align:right is ignored in the resulting note!

Graham

Re: Data References ~ Evernote ~ Research Notes

Posted: 08 Jan 2021 12:29
by tatewise
I suspect you still have </th> instead of </td> in these lines:

Code: Select all

	-- Alignment <align="r"> <align="c"> <align="j">
	strLine, intAny = strLine:gsub('(..-)<align="r">(.-)</td>','%1<span style="text-align:right;">%2</span></td>') 
	intSum = intSum + intAny
	strLine, intAny = strLine:gsub('(..-)<align="c">(.-)</td>','%1<span style="text-align:center;">%2</span></td>') 
	intSum = intSum + intAny
	strLine, intAny = strLine:gsub('(..-)<align="j">(.-)</td>','%1<span style="text-align:justify;">%2</span></td>') 
	intSum = intSum + intAny
I have tested it and it produces the span structure you produced by hand.
Maybe Evernote does not support alignment in table cells?
Try creating a table in Evernote with alignment in cells and export in HTML format to discover what it likes.

Re: Data References ~ Evernote ~ Research Notes

Posted: 09 Jan 2021 09:43
by gsward
I think I have unravelled Evernote's handling of alignment in tables.

It doesn't recognise <span style="text-align:right;"> although it does preserve it in the import and even exporting it again.

It handles alignment like this <td style="text-align:right;"> so I have modified the Alignment section. I have removed the lines like:

Code: Select all

strLine, intAny = strLine:gsub('(..-)<align="r">(.-)</td>','%1<span style="text-align:right;">%2</span></td>')
and replaced with

Code: Select all

strLine, intAny = strLine:gsub('<td>%s-<align="r">(.-)</td>','<td style="text-align:right;">%1</td>')
Does my new pattern matching <td>%s-<align="r">(.-)</td> look OK?

I have also taken out these lines but not sure what the impact is, do I need them?

Code: Select all

strLine, intAny = strLine:gsub('<align="r">(.-)$','<span style="text-align:right;">%1</span>')
Thanks,

Graham

Re: Data References ~ Evernote ~ Research Notes

Posted: 09 Jan 2021 10:43
by tatewise
Good detective work! I might adopt that table alignment form in my Export Gedcom File plugin. The pattern is OK.

You do need those other three strLine:gsub('<align="r">(.-)$'... lines for alignment of whole lines outside tables.

Re: Data References ~ Evernote ~ Research Notes

Posted: 09 Jan 2021 12:00
by gsward
Here is the latest version 0.9 which I have also added to the Wiki page.

Please be aware that Evernote uses some non-standard conventions with its own XML format (ENML). The product is also currently supporting two versions 6.x and 10.x. You need to have version 6.x installed for this plugin to work as the new version does not (yet) support import folders.

Graham

evernote_to_do.fh_lua
(27.57 KiB) Downloaded 94 times
All times are UTC
Page 2 of 2
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/