* FTP Website Manager

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

Already tried - that no luck

Try ftp://users.freepages.rootsweb.com in your browser.
It takes you correctly to the logon screen.

So Site, User and Password are correct on that route.

Does LUA offer any form of trace?

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

No log that I am aware of, but I am still learning how it works.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

Have you seen this
http://www.askdavetaylor.com/why_do_my_ ... freeze_up/

Not an exact match as I think it's a Linux environment
But it looks as though we need PASV mode as in IPSWITCH
Don't know how to do that in LUA

Do you have access to the source of ftp.get ?

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Yes, I have already started looking at that.
Much of the functionality above the fundamental socket layer is implemented in Lua.
Checkout C:\ProgramData\Calico Pie\Family Historian\Plugins\socket
That contains several .lua modules including ftp.lua.

What I have done is copy ftp.lua to C:\ProgramData\Calico Pie\Family Historian\Plugins\ftp.fh_lua.
It then appears in the FH Plugins window and can be easily inspected.
It does appear to perform a PASV command as part of the ftp.get().

I have gone one step further and inserted that ftp.fh_lua script into the FTP Website Manager script, so it can be run in debug mode.
It must be inserted just before the final main() statement.
Then its module("socket.ftp") statement must be removed from near the start of the ftp.fh_lua script.
Also the ftp.put() and ftp.get() must loose their ftp. prefix.

Then with break points on those two statements the put and get functions can be traced.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

I'm not sure I understand this.

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

You asked about the source script of ftp.get.
This is in the C:\ProgramData\Calico Pie\Family Historian\Plugins\socket\ftp.lua file on your PC.
i.e. It is written in Lua just like Plugins, so we can Edit/Debug it just as you've been doing with FTP Website Manager.

However, we have to convert it from a socket Lua module to an FH Plugin script.
That is what the copy ftp.lua to C:\ProgramData\Calico Pie\Family Historian\Plugins\ftp.fh_lua achieves.
It then appears in the FH Tools > Plugins window with the name ftp, and you can Edit it to see the source code.

If you do that, right at the end of this ftp Plugin script is the get entry.
If the parameter is a "string" it calls sget(gett) but our Plugin uses a table so it calls tget(gett).
This is a local function a few lines further up, which opens the host socket, and performs greet, login, type, pasv, receive, quit & close.

My last paragraph explains how to copy & paste that ftp Plugin script into the FTP Website Manager script, so it can be run in debug mode.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

I have taken the socket library modules one step further and incorporated a very basic trace log.

Install the attached variant of the Plugin and run it in Edit/Debug mode as usual.

You will need to set up the FTP Settings from scratch.

When you Exit from the Plugin, the debug pane lower left should display the trace log.
Last edited by tatewise on 11 May 2023 14:10, edited 1 time in total.
Reason: Attachment FTP Website Manager Trace.fh_lua deleted ~ ask Mike Tate (tatewise) if you need a copy
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

Here is the trace - I've erased the password

Connecting: users.freepages.rootsweb.com Port: 21
Response: 220 (vsFTPd 2.0.5)
Command: user billdeverell
Response: 331 Please specify the password.
Command: pass xxxxxxxxxx
Response: 230 Login successful.
Command: type a
Response: 200 Switching to ASCII mode.
Command: pasv
Response: closed
Plugin has completed

What does the response "closed" mean? There's no numeric code.

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Good question. When I run the Plugin the trace I get is shown below. So it seems that rootsweb does not like the pasv command. I could add some extra tracing after that command to see what is happening.

Connecting: http://www.users.freenetname.co.uk Port: 21
Response: 220 ProFTPD Server [freenetname.co.uk]
Command: user miketate
Response: 331 Password required for miketate
Command: pass **********
Response: 230 User miketate logged in
Command: type a
Response: 200 Type set to A
Command: pasv
Response: 227 Entering Passive Mode (212,159,9,151,158,32)
Command: NLST public_html/NewFolder
Response: 150 Opening ASCII mode data connection for file list
Response: 226 Transfer complete
Command: quit
Response: 221 Goodbye.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

I may have cracked it.
I changed pasv to PASV
Trace now is

Connecting: users.freepages.rootsweb.com Port: 21
Response: 220 (vsFTPd 2.0.5)
Command: user billdeverell
Response: 331 Please specify the password.
Command: pass xxxxxxxxx
Response: 230 Login successful.
Command: type a
Response: 200 Switching to ASCII mode.
Command: PASV
Response: 227 Entering Passive Mode (66,43,28,143,248,203)
Command: NLST genealogy_html/test
Response: 150 Here comes the directory listing.
Response: 226 Directory send OK.
Command: quit
Response: 221 Goodbye.
Connecting: users.freepages.rootsweb.com Port: 21
Response: 220 (vsFTPd 2.0.5)
Command: user billdeverell
Response: 331 Please specify the password.
Command: pass xxxxxxxxx
Response: 230 Login successful.
Command: type a
Response: 200 Switching to ASCII mode.
Command: PASV
Response: 227 Entering Passive Mode (66,43,28,143,176,172)
Command: STOR genealogy_html/test//index.html
Response: 553 Could not create file.
Plugin has completed

Not sure why the STOR is using double / before the file name.

The RFC standard says the commands are not case sensitive - obviously that is down to implementation at the server.

Bill
Last edited by Bill Deverell on 02 Feb 2015 10:28, edited 1 time in total.
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

Didn't cross out all the passwords.
Any way you can delete that bit from the web.

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Not sure of the best way forward.
The "pasv" command is embedded within the ftp.lua library module downloaded to the C:\ProgramData\Calico Pie\Family Historian\Plugins\socket\ folder.
You are only able to edit that, because I have unbundled the module into the Plugin script for debugging purposes.

The correct solution is for rootsweb to make the server case insensitive.

I might be able to build a workaround into the Plugin script to modify the ftp.lua library module after it is downloaded.

The double / is a bug in the Plugin for which I already have a correction.

However, just to add some icing to the cake, there is another problem with the ftp.lua library module.
If the Plugin needs to delete files from the server when they no longer exist in the Source Folder the "DELE" command is mishandled.
It results in an error message saying something like filename.type: File not found for the deleted file.

I have a workaround for that too.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

I wait to see your workaround.

Re Upper Case - I see that some of the other commands are already in upper case !!

Thanks

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Yes, the NLST, STOR & DELE commands originate in the Plugin and passed through to ftp.get or ftp.put.
If I really wanted to be consistent, the Plugin should use lower case for those commands too!

I believe the attached Plugin has all the fixes and workarounds that I am aware of.

For the ftp.lua library module workarounds to be applied, the socket library must get downloaded again.
Therefore, before running the Plugin, you must delete the existing socket library as follows.
Navigate to C:\ProgramData\Calico Pie\Family Historian\Plugins\ and delete the socket folder and its contents.
Take care not to delete anything else!
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Capture.JPG
Capture.JPG (21.52 KiB) Viewed 9052 times
Mike

Done and tested.

I get this error msg - see attachment
It then tries to process the other folders in the from folder.

Is the trace still there?

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

No the trace is not possible when using the standard socket library.
However, I suspect the problem is to do with the folder Test on the server, that I suspect no longer exists in your PC folder.
So the Plugin is trying to delete it, but is not allowed to.

I will have look to see if there is a Plugin solution.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

The folder Test is still there on Host and Server.
It's one of 3 folders at that level in both places.
The other 2 are large - containing trees.
The folder Test contains a single item - index.html in both places.

Bill
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

I've inspected the site with IPSwitch.

I think this is what happened in the Plugin.
The STOR operation to upload index.html didn't put it in Test.
It stored it at the next level up
and then tried to delete the 3 folders at that level !
It claims it updated 4 items

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Yes, on closer inspection it does not matter whether folder exists on PC or not.

As stated in the Plugin Help, it is designed to update FH websites (that have no subfolders) and so subfolders are not supported.

However, I have a workaround to ignore them where the subfolder exists on both server and PC.

They only way to ignore subfolders on the server alone is to assume they have no file type, which is consistent with FH website uploading because all website files do have file types such as html, css, js, jpg, png, etc.

Try the attached updated Plugin.

P.S.
The STOR operation will upload to whatever FTP Folder you specify in the Plugin Settings.
As defined by the From: and To: entries at top of Plugin window.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

That improves it a bit.
It only transferred one file but to the level above the folder Test.

I wonder if the folder needs a trailing /

I need to keep separate folders - one for each tree.

Bill
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

Sorry got that wrong.
I'd lost the sub folder on the settings which is why before it went into the wrong level.

When I ran it again I got the attached.
On inspection with IPSwitch nothing had been transferred.

So the STOR doesn't like the subfolder.

I think I'll try it with the Debug

Bill
Attachments
Capture.JPG
Capture.JPG (20.46 KiB) Viewed 9027 times
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

Cracked it.
The folder name is case sensitive.
test should have a capital T.
It did have in previous tests but lost it when I retyped it today.

Thanks for your help.

When my PC is going to be idle for 6 hours I'll try it with a live website
so that the following time there should be less to change.

Bill
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Bill, that is excellent, but I am not impressed by the rootsweb FTP server implementation.

Firstly, its FTP command recognition is case sensitive (at least for PASV).
It is imperative that is reported to rootsweb, because we have had to 'patch' the standard Lua FTP library as a workaround that should not be permanent.

Secondly, it is normal for file/folder names to be case sensitive, but it should report something like:
"450 genealogy_html/test: No such file or directory"
Rather than the red herring of "553 Could not create file" which suggests a permissions problem, not a naming problem.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: FTP Website Manager

Post by tatewise »

Bill, I have added the debug trace feature to the socket library.

Install the attached Plugin, and run it in Edit/Debug mode, then on Exit there should be a trace in the lower left pane as before.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Bill Deverell
Diamond
Posts: 89
Joined: 19 Dec 2013 21:10
Family Historian: V6

Re: FTP Website Manager

Post by Bill Deverell »

Mike

I'll try the version with the trace later.
A trace is always good to have available if anybody else has problems.

Below is text of E-mail I sent to Rootsweb.
You'd think that a standard that was written in 1985 would be well known by now.

Bill

E-mail to Rootsweb

" I have had a lot of problems using a new FTP client in my Family Historian software
and after intensive testing found that the problems were actually caused by an incorrect FTP implementation on your servers.

Basically your servers are expecting FTP commands to be in upper case only
The FTP standard specifies that commands are not to be case sensitive.

The FTP client is implemented using the standard LUA FTP library which had to be modified to force uppercase commands in order to make your server accept the commands.

I also found a 2nd problem caused by incorrect reporting of an error on my part where I had used the wrong case in a folder name.
it is normal for file/folder names to be case sensitive, but it should report something like:
"450 genealogy_html/test: No such file or directory"
Rather than the red herring of "553 Could not create file" which suggests a permissions problem, not a naming problem.
(in my case the folder name should have been genealogy_html/Test

In my case the problems have been solved but there may be other users who will hit the same problem. "
Post Reply