* Text from source - tables in reports

[arthurk]

I know this subject has come up from time to time, but I couldn't find a previous discussion to add these thoughts to. (It doesn't really help when the Advanced Search gives this result: "The following words in your search query were ignored because they are too common words: text source from.")

I don't produce many reports and have only recently started adding census transcripts in the tables that v.7 introduced (generic sources, autotext). The ability to shrink wide tables helps a bit, but when that happens, a single word that's too long for a cell gets split on to two lines in order to fit. This does not look good, so I have a couple of thoughts about this:

1. Has anyone tried to calculate the optimum width of a table in whatever units FH uses, so as to take up the available space and no more, and thus avoid the need for shrinking? And, related to this: where can custom autotext templates be found, in order to view or edit this value?

What I'm thinking is, if I can accept the constraints of this overall width, whatever I put in a table would, I hope, be reproduced in exactly the same way in a report.

2. How does font size come into this? The font setting in the autotext note window determines what is seen in that window, but how does it transfer to a paper report - if it does?

3. Regardless of the answer to that question, would it be helpful in report options to be able to specify the font for Sources separately, or even just for tables within sources? With a smaller font the table takes up less room, and you're less likely to need to shrink it.

My font is currently set at size 10, and I found that reducing it to 8 improved the look of the tables - and in fact I'd be happy to use that for the whole of the Sources section. However, Sources currently share a font setting with the main content of a report (in Section Data), and I think 8 is too small for that. Separating these font settings would, I think, be a useful enhancement. Is that worth proposing for the wish list?

[NickWalker]

Ancestral Sources does make it easier to reduce table column widths but ultimately some of the 20th century census tables just won't fit onto a page width because there are too many columns and too much data. There is a post I made on the wish list request forum that refers to this and may be of interest.

[arthurk]

Thanks, Nick - quite a bit of food for thought in that thread.

So far I've been trying to copy the layout of certificates and census returns, including long lines of text or line feeds, but I'm aware that long lines in particular are part of the problem. I'm beginning to wonder whether in the interests of presentation it would be better to treat my transcription as a string of text that breaks between words as necessary to fit in cells, even if the layout doesn't precisely match the original. (With certificates, you can't be sure anyway if the layout matches what was in the register - and since marriages were recorded in duplicate registers, even if the words were the same, I don't think the layout had to be.)

In some cases I'm already adapting the data to a more compact format: (a) in my 1911 census table I've replaced the 4 columns for length of marriage and details of children with one headed "Mar/Chn", to be completed in the format 10/5/3/2, with a note below the table to explain this; (b) in 1921, when I get there, I'll be replacing the multiple child columns on the right with one headed "Children under 16 (& ages)"; (c) I include Employer/Worker information in the Employment column rather than a separate one; (d) my column headings are mostly abbreviated versions of the originals.

Whether this kind of thing can be carried through to produce a table that will never need to shrink is what I'm really asking. I feel relatively fortunate that I only have a handful of non-UK censuses, but even before this transcription exercise I wasn't recording every last detail from some of them because I didn't feel it had much value for my purposes. Maybe one way to go with these would be to record the major details (the same kind of thing as on a UK census), and have a note underneath the table saying there's more information on the original form, which can be found at xyz website.

[Mark1834]

For me, the question is "what will I do with the reports?" I transcribe UK census entries as simple plain text, such as "John Smith, Head, Mar, 43, M, Layabout, b Somewhere", one row per person, blank fields omitted with no double commas. It's fully searchable, and I've got the image as well if I want to check anything. But the only reports I produce are data summary reports that are useful for proofreading or getting an easy overview of the data I have on a particular person or family.

As such, I get no value from formatted text, but appreciate if somebody wants the final output to be of a higher presentation standard, their priorities will differ.

[NickWalker]

Yes I rarely use reports either, but do find it much easier to read source text which uses a rich-text table as the all the columns line up. And of course AS generates that for me so it's no more extra work. The image is there when I need it but it's quicker to read a transcription.

[arthurk]

For me, the question is "what will I do with the reports?"....

And 99% of the time I'm not really worried about how tables appear in reports either. It's just that recently I needed to produce a pdf report to send to someone else and was a bit embarrassed about how they appeared so I ended up not including text from source at all.

This was admittedly before I updated to v.7.0.18 with its option for splitting tables - but even with that available, it's been niggling at me that there might be/ought to be a way to produce tables that will fit on a page without needing to be split or shrunk, because they look so much neater.

[davidf]

For me, the question is "what will I do with the reports?" I transcribe UK census entries as simple plain text, such as "John Smith, Head, Mar, 43, M, Layabout, b Somewhere", one row per person, blank fields omitted with no double commas. It's fully searchable, and I've got the image as well if I want to check anything. But the only reports I produce are data summary reports that are useful for proofreading or getting an easy overview of the data I have on a particular person or family.

Perhaps you might store the transcript of household details as JSON-data - I suspect FH functions might be persuaded to be able to pull out specific data items? We could even have a plug-in JSON-viewer!

[arthurk]

From my original post:

1. Has anyone tried to calculate the optimum width of a table in whatever units FH uses, so as to take up the available space and no more, and thus avoid the need for shrinking? And, related to this: where can custom autotext templates be found, in order to view or edit this value?

I've now found them, in a hidden folder - C:\ProgramData\Calico Pie\Family Historian\Autotext\Text from Source

If you adjust the table's layout the column width figures may of course vary; the actual ones in use can be seen in the project's Gedcom file as part of the source definition. But does anyone know what those units are, and how they relate to things like inches or centimetres, or even pixels?

[tatewise]

In FH use Tools > Manage Autotext... rather than digging in ProgramData.

[arthurk]

In FH use Tools > Manage Autotext... rather than digging in ProgramData.

Where in Manage Autotext can I view a table's column widths? This is what I'm missing.

[tatewise]

Use Tools > Manage Autotext... then [+] Text from Source, then [+] any of Canada, UK or USA.
Select any Census option and click Edit... and OK warning to see Rich Text tables, etc.
Alternatively, use Clone... to avoid editing masters and use Edit... so you can adjust the width of the table columns.
Is that what you were looking for?

[ColeValleyGirl]

Mike,

I suspect OP wants to see the raw text underneath the rich text, that you can can get at by editing the file directly, or (for Notes created from autotext) by using the method here: https://fhug.org.uk/forum/viewtopic.php ... 94#p127234 to get this view

Screenshot 2022-12-11 083123.jpg (61.75 KiB) Viewed 3793 times

It doesn't seem possible to do this from the Manage AutoText dialog, so the best I can suggest is to create a Note Record, insert the Autotext you want to inspect there, and delete the Note record once you're finished with it (maybe saving it as new Autotext first if you've got something you're happy with).

Maybe somebody else has a better idea.

[davidf]

Is it an error in the first line of the above example that the markup is interwoven rather than nested? Certainly in HTML/XML/XHTML that line would fail a syntax checker

[ColeValleyGirl]

Is it an error in the first line of the above example that the markup is interwoven rather than nested? Certainly in HTML/XML/XHTML that line would fail a syntax checker

That's exactly as supplied with FH by CP (UK 1841 census text from source) -- no error.

[tatewise]

Yes, Rich Text format does not seem to require the tags to be in balanced nested pairs.
That makes writing code (such as in plugins) particularly complicated.
The GedSite author made the same comments when handling FH GEDCOM.

[arthurk]

Mike,

I suspect OP wants to see the raw text underneath the rich text, that you can can get at by editing the file directly, or (for Notes created from autotext) by using the method here: https://fhug.org.uk/forum/viewtopic.php ... 94#p127234 to get this view

Screenshot 2022-12-11 083123.jpg

It doesn't seem possible to do this from the Manage AutoText dialog, so the best I can suggest is to create a Note Record, insert the Autotext you want to inspect there, and delete the Note record once you're finished with it (maybe saving it as new Autotext first if you've got something you're happy with).

Maybe somebody else has a better idea.

Thank you, Helen, that is what I was looking for. (I'd remembered reading about the method you've referred to when it was first posted but didn't find the link. I've now bookmarked it.)

I'm now hoping to experiment a bit to see if I can come up with some useful numbers to guide table designers. I'll report back in due course.

(Incidentally, until exploring just now I hadn't grasped the range of possibilities that Autotext offers. It took me a while to find how to insert it into a Note (in the rich text editing window, on the gear wheel drop-down), but I then realised that you can define any number of snippets of frequently-used text and have them available to insert. Typically, I can't think of where I might use these at present, but it's nice to know that it could be done.)

[davidf]

Yes, Rich Text format does not seem to require the tags to be in balanced nested pairs.
That makes writing code (such as in plugins) particularly complicated.

"does not require" either means that it genuinely does not matter, or that it has a reasonable tolerance of dodgy syntax. It may be when the syntax gets too dodgy that we start to have problems.

But is this RTF (the Microsoft proprietary format) - which I associate with braces and back slashes and control words {\word} - not for hand coding!? (I understand late versions of RTF - 1.9.1 ? - tolerated some XML syntax) It looks like a different form of markup - to me closer to HTML (even XML so possibly a subset of RTF 1.9.1?) than "pure" RTF.

Presumably FH is using a third party library to "implement Rich Text", so any oddities are down to the Library used rather than CP?

[ColeValleyGirl]

But is this RTF (the Microsoft proprietary format) - which I associate with braces and back slashes and control words {\word} - not for hand coding!? (I understand late versions of RTF - 1.9.1 ? - tolerated some XML syntax) It looks like a different form of markup - to me closer to HTML (even XML so possibly a subset of RTF 1.9.1?) than "pure" RTF.

No. Syntax documentation is here

[tatewise]

Yes, but I thought the way Nick talks about handling Rich Text in AS is that some Windows/Microsoft standard library is involved, so the syntax must be based on some public documentation associated with that standard library, and I assume FH uses the same library.
Anyway, the FH Help page says nothing about codes needing to be in balanced nested pairs, but various examples that appear in Rich Text Notes, Autotext, etc, suggest that is not a requirement.

[davidf]

No. Syntax documentation is here

Ah!

So, FH "Rich Text format" is not Microsoft Rich Text Format (the case of the initial letter of format being rather vital!). It is FTF!

'FTF' stands for 'Family Historian Text Format'.

Probably fun to write a "Rich Text" editor from scratch, but what is wrong with something like XML or almost any other established markup language!?

[ColeValleyGirl]

Yes, but I thought the way Nick talks about handling Rich Text in AS is that some Windows/Microsoft standard library is involved, so the syntax must be based on some public documentation associated with that standard library, and I assume FH uses the same library.

You may be right, but I'm not sure it's a safe assumption. Msoft are unlikely to provide much if any support for markdown formats outside the mainstream.

I'd be interested to hear how Nick generates the FTF output.

what is wrong with something like XML or almost any other established markup language!

Are the internal links a challenge, I wonder?

[NickWalker]

I'm not sure that HTML does require pairs to match in the way you've suggested, or at least this seems to work perfectly well when rendered by Edge/Chrome:

Code: Select all

<html>
<b>Hello <u>there</b> how are you?</u>
</html>

And actually this forum has no problem with this either:

Code: Select all

 [b]Hello [u]there[/b] how are you?[/u]

Ancestral Sources (and I assume Family Historian) makes use of a dll control that provides the actual text box that renders richtext format. It handles the editing of the text, displaying tables, allowing table columns to be resized and so on. As a programmer I can extract the rich-text format from that. So the text: Hello there how are you? looks like this in RTF:

Code: Select all

{\rtf1\ansi\ansicpg1252\deff0\nouicompat\deflang2057{\fonttbl{\f0\fnil\fcharset0 Tahoma;}{\f1\fnil Tahoma;}}
{\*\generator Riched20 10.0.22000}\viewkind4\uc1 
\pard\b\f0\fs20 Hello \ul there \b0 how are you?\ulnone\f1\par
\par
}

I then have to parse this text to turn it into the Family Historian FTF which was not easy to do, particularly when dealing with the way fonts are handled and hyperlinks, etc. I also created another parser that can convert from FTF back to Richtext.

My assumption is that Calico Pie tried to make FTF be very similar to HTML. I do wonder why they didn't use an HTML wysiwyg control rather than a rich-text control but this may be due to them not being able to find anything suitable.

[tatewise]

It certainly looks more like the Rich Text Format specifications I've found.

The conversion between that MS rich-text and FH Rich Text looks quite complex in places

[davidf]

I'm not sure that HTML does require pairs to match in the way you've suggested, or at least this seems to work perfectly well when rendered by Edge/Chrome:

Browsers are pretty tolerant of malformed HTML - mainly because the standards kept changing - particularly with the "requirements" for "closing tags" (e.g. </p>) and the option for self-closing tags (e.g. <hr />, <br />) to maintain compatibility with the fussier XML and XHTML standards, which never gained widespread acceptance - possibly because they were too fussy and hand-coders (I remember them) used to make mistakes!

[arthurk]

Trying desperately to keep my thread on track, I've now done some experiments relating to tables and the way the average user might want to use them in text from source. But first, thank you Helen for

No. Syntax documentation is here

From that I now know that the columns in a table (in text from source etc) are measured in twips. For those like me who didn't know, a twip is 1/20 of a point, which itself is 1/72 of an inch. In metric, 1 twip = 0.01764mm approx.

Table size

The number of twips you have available for a table will depend on the size and orientation of your paper, and the size of margins etc. By trial and error I discovered that with A4 portrait paper (L & R margins 1.27 cm; indent 0.76 cm) the optimum size for a table was 9500 twips. In the rich text editing window, for me that lined up with the bottom of the 'Link' icon in the toolbar (see image).

ak-tablewidth.jpg (8.97 KiB) Viewed 3619 times

If you don't want to use the raw text editing window, you should be able to pick a point like that and save it into one of your autotext templates, and never actually worry about how many twips you're using. Anyway, once you've set the overall size of the table it's important not to adjust the right hand edge. Moving column dividers within the table doesn't affect its overall size, nor does anything typed into the table.

Fonts

If you use the default font settings, the editing window uses whatever is set as default for Notes (mine is Tahoma 8pt), while a report uses whatever is in Section Data (mine is Times New Roman 10pt). Choosing a different font for the table (or the whole of the text from source) means that in a report you can differentiate a source's text from its title, and, if it's more compact, make maximum use of the limited space available.

Finding the best font is a bit hit and miss, and depends on which ones you have available. It's also necessary to test them in a sample report, as I found that some of the ones in the editing window dropdown weren't available in a report.

In the normal editing window you can't set a font size smaller than the current default for Notes: for example, with a default of 8pt you can't choose 6pt or 7pt in the expectation that it will carry across to the report and reduce a default 10pt there to 8pt or 9pt. It is possible to set a smaller size in the raw text editing window, which is then applied to the report, but for most users it's probably better to use the dropdown to pick something suitable.

The fonts which I have which seemed to come out smaller than Times New Roman were these (in fact some are so small that rather than the editing window's default 8pt size I set them at 9pt - these are marked (9)): Agency FB (9); Centaur; DejaVu Serif Condensed; Gabriola; Microsoft Himalaya (9); Microsoft Uighur (9); Onyx; Perpetua; Scheherazade; Sitka Display.

Whatever font you use, what you see in the editing window may well transfer to the report with different line breaks etc, so it's a good idea to check the output. For good presentation the most important thing is to get the headings looking right, so it will probably be necessary to adjust the column widths. Any text lower down that's too long will simply wrap on to a new line, but a word that's too long for its cell will break somewhere, so checking first lets you set the break yourself and hyphenate it in the best place.

Those who have read my earlier posts may realise that I am giving up on the idea of trying to slavishly replicate the exact layout of source documents; it might be possible sometimes, but I now feel that having all the important text easily visible is better than allowing a table to overflow the page.

I can't guarantee that every table for every kind of source can be made to fit on a page, but being aware of the constraints should make it easier to make them as clear, concise, complete and consistent as possible.