* Data References ~ Evernote ~ Research Notes

[gsward]

I don't know whether this is a missing piece of functionality in V7.

I am trying to look at modifying the Evernote plugin to work with Research Notes rather than individuals. It feels like identifying the required field names for the _ROTN tag is a bit hit or miss. So I turned to try and use the Data Reference accessed from Configure Columns for Record lists. I get as far as dialog to Edit the column details but the "<< Insert Data Ref ..." button does not work. Am I missing something or is FH?

Maybe it is a different question I need to ask as what I need to be able to do is to extract the RTF for the Research Note to pull it into my plugin so it can be passed to Evernote.

Graham

[davepacey]

I have already submitted a ticket to calico pie about this issue.

[gsward]

Thanks. I should not have asked two questions in the same post...

Does anyone have an insight into the data references for RTF formatted notes?

Thanks,

Graham

[tatewise]

Yes, it is %_RNOT.TEXT% for the Research Notes Text field.

Depending on what Evernote needs you may need to handle the field in various ways.

In the Tools > Plugins > How to Write Plugins help pages see The Family Historian API + Rich Text Syntax.

[gsward]

Yes, I had looked at that page. It does not exactly tell me how to access the RTF (or FTF and they define it). %_RNOT.TEXT% is clearly the right starting point but it only returns the first line of the RTF block (the Research Note text area.)

Graham

[tatewise]

Why do you say %_RNOT.TEXT% only returns the first line?
Where are you looking?

If you are looking in the Plugin editor/debugger then the Variables pane bottom right only shows a summary.
Double-click any variable to see its full contents.

In the Plugin help pages check the Function Index that has a few fhFtf... and fhNew/Get/Set...RichText functions, and also under Objects + Rich Text there are all the Rich Text methods.

[gsward]

In the plugin there is a section of XML to generate the content for the export.

There is a block as follows:

Code: Select all

<h3>Comment</h3>
<div>{INDI._RNOT[1]>TEXT| | }</div>

This finds the correct field but only returns the text of the first line. Clearly, I need to use one of these FTF functions to extract the Rich Text but I cannot see how to get from INDI._RNOT[1]>TEXT to something that incorporates fhGetValueAsRichText from the examples given.

[tatewise]

Lines such as <div>{INDI._RNOT[1]>TEXT| | }</div> are just a 'pattern' that is processed later.

The actual data retrieval is performed inside the function createNote(ptr,sComment,sTags,sTitle) within its internal local function replacefields(field) which splits each component in {brackets} into dataref, prefix & suffix.

It is the dataref that sets up dataPtr which in your case points to INDI._RNOT>TEXT

It then checks the data class and for longtext uses fhGetValueAsText
Otherwise, it just uses fhGetDisplayText which produces the one line you are getting.

The data class of INDI._RNOT>TEXT is richtext which is new for FH V7.
So try this:

Code: Select all

        if dataPtr:IsNotNull() then
            if fhGetDataClass(dataPtr) == 'longtext' then
            newData = prefix..StrCP1252_XML(fhGetValueAsText(dataPtr))..suffix
            elseif fhGetDataClass(dataPtr) == 'richtext' then
            newData = prefix..StrCP1252_XML(fhGetValueAsRICHText(dataPtr))..suffix
            else   
            newData = prefix..StrCP1252_XML(fhGetDisplayText(dataPtr))..suffix
            end
        end

[gsward]

So I gave this a try. Is there maybe some global setting required for lua to make this work? I get the error:

Code: Select all

... attempt to call a nil value (global 'fhGetValueAsRICHText')

[tatewise]

Sorry, my typo, should be fhGetValueAsRichText
Unfortunately, you have to get the case of such functions exactly right!
That is the nature of Lua which is very case sensitive.

Beware that since it returns a special rich text data type it may be unsuitable for embedding in the Evernote file.
So you may have to use the :GetText() method too but whether that then produces rich text in Evernote I don't know.

[gsward]

OK, that gets that error out of the way, but now it takes a disliking to 'gsub' in:

Code: Select all

function StrEncode(strText,strPattern)
    strText = (strText or ""):gsub(strPattern,TblCP1252)
    return strText
end -- function StrEncode

Code: Select all

... attempt to call a nil value (method 'gsub')

[tatewise]

As I said, it is a special rich text data type and not plain text, which gsub requires.
( I had the same problem in my Search and Replace plugin for rich text fields. )

So try using the :GetText() method:

newData = prefix..StrCP1252_XML(fhGetValueAsRichText(dataPtr):GetText())..suffix

[gsward]

Well, that works. Thank you.

However, the parts of the Research Note that were in Rich Text have been converted to raw text i.e. like html source with tags.

Are there any other alternative methods that might preserve the formatting? Like being able to convert from RFT to HTML?

If I use GetPlainText() it is better but of course, ALL the formatting is lost, especially any tables which do happen to be used in Research Notes.

[tatewise]

Without knowing how Evernote rich text format is encoded we are stabbing in the dark.

I went through a similar exercise with the Export Gedcom File plugin to determine what rich text word-processing format each target product supported. Some had their own format. Some used a subset of HTML5 codes.
However, the simple <b>bold</b> and <i>italic</i> and <u>underline</u> codes are standard HTML.
So if Evernote does not even recognise those simple codes then any more sophisticated HTML is unlikely to help.
Sorry I cannot be more helpful.

[gsward]

OK, I don't want to drag you down a rabbit hole, but here is what I have found.

URLs get exported from FH as
<web="https://www.ancestry.com/","Ancestry.com">

rather than
<a href="https://www.ancestry.com/">Ancestry.com</a>

Tables as:

<table="7865|1715|1265">
<row> Task Description | Status | Date Completed </row>
<row> <rec=1,"Elizabeth MAYES"> | | </row>
<row> | | </row>
<row> | | </row>
<row> | | </row>
<row> | | </row>
<row> | | </row>
</table>

Which is really just an XML snippet. This is precisely the FTF formats as described in the FH Plugin Help. Ideally, we need a function or method to convert FTF to HTML.

[tatewise]

I have Lua script that converts those FTF codes to HTML that I use in the Export Gedcom File plugin.

But I suggest you take a step back and instead of those more complex formats just use bold italic and underscore in your Research Note, then try and send that to Evernote using your plugin, because the FTF and HTML codes are the same.

If that is accepted by Evernote and displays the formatted text then we know at least those simple HTML codes work.
So we can then progress to the more complex HTML codes with some degree of confidence.

That is similar to what I did for each product that my plugin supports.

[gsward]

Yes, this is interesting. I had started to do this hence I realised the table was not being formatted with conventional html tags.

I have set up a note with examples of bold, underline, italic and strikethrough plus the table. The tags pass through to Evernote OK it seems but gets rendered as <b>First Name</b> (i.e. as text) rather than being interpreted as html, like First Name.

So I exported the note from Evernote, which you can do in various formats, ENEX allows you to move it to another Evernote account, effectively an archive or imported into OneNote, but you can also export as html.

This is where it proves to be interesting. If we open the html as a text file we can see what is going wrong, here is a snippet:

It is being rendered as:

Code: Select all

Description: &lt;b&gt;Grave&lt;/b&gt;<br/>

rather than

Code: Select all

Description: <b>Grave</b><br/>

So why are some characters being passed correctly as < and > others as &lt; &gt; ?

It is the ones from the Research Note that are not playing nicely. Is it a coding issue? It looks like an adaption of the function StrCP1252_XML is required, but we would also need to correct the way it is processing urls.

[tatewise]

Fundamentally, Evernote is not recognising <b>First Name</b> as HTML coding but simply as literal text.
When exported as HTML there has to be a method to distinguish between HTML tags like <b> and literal text <b>.
So < becomes &lt; and > becomes &gt; and all reserved HTML characters are similarly treated so & becomes &amp; and so on.

When the HTML rendering engine sees &lt;b&gt; it displays <b> but when it sees <b> it switches on bold mode.

So until you can discover how Evernote expects rich text to be formatted there is no way forward.

I've just realised that the StrCP1252_XML(...) function call is 'hiding' all those HTML reserved characters as described above, so try removing that. i.e. use newData = prefix..fhGetValueAsRichText(dataPtr):GetText()..suffix
We can worry about URL coding later.

Have you ever reviewed the contents of the temp.enex file created by the plugin?
Have you ever tried manually creating a temp.enex file and opening that with Evernote?
That is what you have to do to explore these scenarios. The plugin suggests temp.enex is an HTML file.

[gsward]

Funnily enough in some of my earlier researches when things were not working at all I started looking at the temp.enex file!

Now when I remove the encoding function StrCP1252_XML and just use newData =prefix..fhGetValueAsRichText(dataPtr):GetText()..suffix it does not import at all, complaining there is no note content.

So my plan is to investigate the differences between the two enex files plus one correctly formatted one I will set up in Evernote and export as enex. I'll report back later.

[gsward]

Evernote uses a form of XHTML they call ENML. From their guide

Applications that wish to store plaintext notes in Evernote must convert newlines into HTML-style blocks. To ensure that notes render correctly across clients, we recommend that you wrap each paragraph in a <div> element. For each blank line, insert a <div> containing a single <br/>. 

Simply each line needs to begin and end with <div> and </div>
Blank lines include <br/>

Code: Select all

<div>Simply each line needs to begin and end with</div>
<div><br/></div>

What's wrong with our transformation?

We are only getting one pair of <div> </div> at the beginning and end of the block that comes from the Research Note not for each line.

Here is a summary of how the tags get transformed. (See the attached table)

FH_Evernote.xlsx

(5.41 KiB) Downloaded 101 times

[tatewise]

Presumably, you have added <div> and </div> to the plugin template, as they don't exist in the original 'Evernote To Do'.
I imagine the original plugin only ever exported plain text, so did not need them.

So I suggest you compose a temp.enex file by hand with the necessary <div> </div> and simple <b> </b> codes, etc.
Once that is accepted by Evernote, then the plugin template can be adjusted to fit the rules.

If I have understood correctly, it means that both the 'longtext' and 'richtext' versions of newData will need to loop on each line to surround with the <div> and </div> tags and insert </br> for blank lines.

In addition, the 'richtext' newData will need converting from FTF to HTML but than can come later.
Let's just get <b> </b> and <i> </i> and <u> </u> working first.

[gsward]

Yes, I had added the <div> </div> to the plugin template. It might be redundant for the 'richtext' if subsequently they are added by some loop to each line after the FTF conversion.

In my testing I have found that a heading tags e.g. <h3> </h3> don't actually need the <div> and </div> wrapper, but if it is there it is not a problem. It is essential though with </br>, i.e. this has to be <div></br></div> to work.

I also tested the unordered list as this is useful for laying out BMD facts and the table as these are used in the Research Notes. Tables are not perfect as Evernote has limitations in table formatting about which there is a plethora of discussions and workarounds on Evernote forums.

I have zipped the test enex file as it was blocked as suspicious and my current version of the plugin.

test.zip

(643 Bytes) Downloaded 52 times

Evernote_to_do_INDI.fh_lua

(21.68 KiB) Downloaded 48 times

[tatewise]

I presume that test.enex file loads into Evernote which renders the contents correctly.

Does that apply to <div><s>strikethrough text</s></div> as I thought it needed <div><strike>strikethrough text</strike></div> ?

A possible problem I foresee is where <b>bold text</b> (and other formats) span several lines.
i.e. does this work?
<div><b>start bold text</div>
<div>continued bold text</div>
<div>end of bold text</b></div>

or will the <b> and </b> tags have to be repeated on every line?

What is happening about lines 61 & 62 in the plugin as the commented out line and the active line are the same?

Code: Select all

--            newData = prefix..StrCP1252_XML(fhGetValueAsRichText(dataPtr):GetText())..suffix
              newData = prefix..StrCP1252_XML(fhGetValueAsRichText(dataPtr):GetText())..suffix

Nevertheless, that seems to be a significant step forward and the plugin can adopt those confirmed formats.

[gsward]

Evernote is not happy with formats spanning several lines. It fails to import a gives an error, like:

Code: Select all

Opening and ending tag mismatch:  b line 9 and div

My duplicated and commented out line 61, is from my testing. It is just a convenient way of "backing up" before making changes, so when we are happy we can just delete it.

On the <s> <strike> question, both actually appear in the ENML standard. It imports <s> OK, but I have found if you do an export from Evernote it uses <strike>! Actually, I now see that <strike> became obsolete in HTML5, so we should stick with <s>.

Thanks.

[tatewise]

I have incorporated the HTML conversions from my Export Gedcom File plugin into your plugin.
It does not yet cope with the problem of formats spanning multiple lines and may need further refinements.
However, see how it deals with the simpler formats to check we are on the right track.