* fhFileUtils creates TEXT files in BINARY

[Mark1834]

I don't know if this is on the list of known issues with fhFileUtils, but I was getting very confused by the way it writes a text file.

The following script takes a list of values, creates a table, and writes that table to file. Standard Lua file i/o in its default text mode and fhSaveTextFile(...) both do the correct translation to Windows text files, with each line terminating in <CR><LF>. However, createTextFile(...) writes a binary file, not text, with just a <LF> line ending. This is identical to the file created with Lua file i/o specifying a binary output.

IMO opinion, this is very confusing, and means that different techniques will be needed to read back a file created with fhFileUtils compared with other standard Windows text files. If I try to add the \r explicitly when creating the contents, createTextFile(...) ignores it anyway and still produces incorrect line endings.

Code: Select all

fhfu = require('fhFileUtils')

F1 = 'V:\\Junk\\JunkLuaText.txt'
F2 = 'V:\\Junk\\JunkLuaBin.txt'
F3 = 'V:\\Junk\\JunkFH.txt'
F4 = 'V:\\Junk\\JunkFHFU.txt'

tblT = {}
table.insert(tblT, 'line1')
table.insert(tblT, 'line2')
table.insert(tblT, 'line3')

filet = io.open(F1, 'w')
io.output(filet)
io.write(table.concat(tblT, '\n') .. '\n')
io.close()

fileb = io.open(F2, 'wb')
io.output(fileb)
io.write(table.concat(tblT, '\n') .. '\n')
io.close()

fhSaveTextFile(F3, table.concat(tblT, '\n') .. '\n')
fhfu.createTextFile(F4, true, true, table.concat(tblT, '\n') .. '\n', 8)

[tatewise]

This is related to Knowledge Base, Libraries, Snippets, Unicode File-paths (20081) in my first post under Load/Save Text File.

The behaviour you describe is presumably a side effect of using ADODB to implement that createTextFile() function.
I wonder how the complementary function readTextFile() behaves, as that also uses ADODB.

However, those two functions plus fileGetContents() and filePutContents() are now largely redundant as they have been overtaken by the FH API fhLoadTextFile() and fhSaveTextFile() functions.
IMO those fhFileUtils functions should be removed or the documentation should refer to the FH API functions that are more well behaved and support full Unicode file paths.

[Mark1834]

The FH version has fewer options, but it does exactly what the documentation says it does so that’s a strong tick in its favour.

[tatewise]

The FH API functions don't have many fewer options.

fhLoadTextFile() does everything that readTextFile() does without the 20Mb size limitation.
By default, it will load ASCII/ANSI/UTF8 text and with the "UTF-16LE" parameter it loads UTF16 text.
Where it says "The file must be in UTF-8 format or UTF-16LE." it is lying, as plain ASCII/ANSI files are OK too.
It supports full Unicode file path names, and if the file does not exist it returns nil which is effectively the same as false.
fileGetContents() loads any kind of file in binary mode using Lua io.open, etc, but does not allow Unicode paths.

fhSaveTextFile() does most things that createTextFile() does.
It does not protect against overwriting existing files and cannot create plain ASCII/ANSI files.
It supports full Unicode file path names, and it returns a boolean to indicate success or failure.
filePutContents() saves any kind of file in binary mode using Lua io.open, etc, but does not allow Unicode paths.

[Mark1834]

Where it says "The file must be in UTF-8 format or UTF-16LE." it is lying, as plain ASCII/ANSI files are OK too.

I would not accuse CP of lying in their documentation. fhLoadTextFile(...) does NOT correctly import extended characters encoded as ANSI. It has to be UTF-8 encoding (same characters, different bytes).

Try it by creating an ANSI-encoded file in something like Notepad++. fhLoadTextFile(..) misreads it, but if the encoding is changed to UTF-8 in Notepad++, it reads correctly.

[tatewise]

You are correct. I should not have included ANSI in the list.
However, neither fhLoadTextFile() nor readTextFile() load extended ANSI characters correctly.
At least fhLoadTextFile() makes those ANSI characters stand out by substituting � which is quite rational.
Whereas, readTextFile() strips the most significant bit and turns them into valid (but wrong) ASCII characters.
I know which I would prefer.

So I still maintain that fhLoadTextFile() does everything that readTextFile() does and more...

[Mark1834]

My preferred endpoint is probably one where fhFileUtils concentrates on file and folder management tools for Unicode (so essentially a replacement for the lfs library), and file I/O is handled by a combination of FH functions and the standard Lua i/o library, which is easy to use and not difficult for a novice author to learn.

That would help avoid relatively unproductive and potentially confusing postings where the preferred option is promoted and the other denigrated.

Where there is overlap between FH functions and libraries, my tendency is to use the former, not least because they have probably been tested more thoroughly, and by coders with detailed knowledge of FH's inner plumbing.

[tatewise]

Yes, I'll go along with that.
Ideally, the fhFileUtils functions should not replicate FH API functions or standard Lua io functions.

One lfs feature missing from fhFileUtils is the lfs.dir() iterative looping mechanism as used in the Code Snippet Directory Tree.
See my first post in Knowledge Base, Libraries, Snippets, Unicode File-paths (20081) under Directory Tree Code Snippet.
An associated problem is managing file Date Modified timestamps via the lfs modification attribute, lfs.touch() and io.time().

One technique I use where both Lua file io functions and Unicode file paths are needed is to use the C:\ProgramData\Calico Pie\Family Historian\Plugin Data\ folder which has no Unicode characters in the file path.
That allows the Lua io functions to manipulate files of any type, then the fsFileUtils or FSO copyFile or moveFile functions can transfer the file to a Unicode file path.