Mike's Map Life Mapping plugin - use?

Cambiz · Post by **Cambiz** » 13 Mar 2012 16:06

Ignore the second part. I ran it again (map this individual) and it still said not enough memory.
However watching the Resource showed me that the when I ran the plugin the memory went from 1804MB in use memory to 2161MB at the max before the Not Enough Memory pop-up.
There was plenty of free memory as I had cleared the standby memory.

email is camichael.fh1@gmail.com as I forgot to add it last time

--
Chris

Post by **tatewise** » 13 Mar 2012 16:21

You say there are no Statistics ringed in blue below, and no Status ringed in green below.
The three mapping buttons should be inactive until the Statistics and Status show something.

After starting the Plugin there should be a Status message initially saying OK and then Loading Individuals list, please wait.

Does the no memory error still happen if you do NOT touch any buttons at all?

Cambiz · Post by **Cambiz** » 13 Mar 2012 16:33

Sorry

Cambiz · Post by **Cambiz** » 13 Mar 2012 16:35

Only happens when I run plugin.

I have removed 1.11GB of files from the directory and will give it another go shortly

Cambiz · Post by **Cambiz** » 13 Mar 2012 19:01

Think I've cracked it.

It worked after I removed the 1400KB _nameindex.html file

Post by **tatewise** » 13 Mar 2012 22:23

The Plugin decides if it is creating map files in a FH website folder by testing for the existence of _nameindex.html and fhstyle.css.
It also obtains the header script and menubar script from _nameindex.html for insertion into the map files.

So with no _nameindex.html it is the same as a non-website folder, and the map files are not linked to the ind and fam files.

But that has given me something to look at, and confirms where I should be looking for the problem.

Post by **tatewise** » 13 Mar 2012 23:23

I am 99% sure I have found the problem.

Would you be prepared to edit the Plugin script to check the fix?
If all goes horribly wrong you can always download the Plugin again from Plugin Store.

Use Tools > Plugins and click the More>> button lower right.
Select Map Life Facts from the Plugin list and click the Edit button near top right.
In the window that opens use Edit > Find and in Find what: enter nameindex.
Ensure Direction is Down and click the Find Next button twice and then click Cancel.
That should find the section of code shown below:

Code: Select all

                        -- Folder contains FH Family Tree CD/Website so map file needs FH headers & FH menubar
                        for strLine in io.lines(strOutputFolder..'_nameindex.html') do
                              strWebPage = strWebPage..strLine..'n' -- Read entire name index HTML file
                        end

Please insert one if statement line using copy & paste so the code becomes:

Code: Select all

                        -- Folder contains FH Family Tree CD/Website so map file needs FH headers & FH menubar
                        for strLine in io.lines(strOutputFolder..'_nameindex.html') do
                              strWebPage = strWebPage..strLine..'n' -- Read entire name index HTML file
                              if strLine:find('<div class=') then break end
                        end

Leading tabs and spaces are unimportant.

Then use File > Save and close the editor window using red X top right.
Finally Close the Plugins window.

Now try running the Plugin with your biggest webpage folder but ensure that _nameindex.html is present.

Cambiz · Post by **Cambiz** » 14 Mar 2012 09:40

Yes, that has worked and I no longer see the 'no more memory' popup.

--
Chris

Post by **tatewise** » 14 Mar 2012 11:47

Excellent - That fix, with some other minor updates will be included in the next version soon.

Post by **tatewise** » 15 Mar 2012 11:05

Version 2.6 is now available to download from the Plugin Store, with the following features.

Fixes the not enough memory error for a large _nameindex.html file.
Opens _nameindex.html file only once per Map... button operation so mapping runs faster.
Popup WARNING! ADVICE! for Geocoder Daily Quota Exceeded now appears when Plot All.., Map All.., or Map Some... button first used per Project, regardless of how many Locations need to be plotted. You can choose to suppress the popup thereafter. The reasoning is that the Daily Quota can be exceeded with less than 2,500 Locations in database, if they are re-plotted more than once within 24 hours.
A few other minor coding style and performance updates.

YouTube · Post by **Jane** » 15 Mar 2012 12:35

Mike, just as an aside if you want load the entire file don't concat the strings, it's much better to load the file in to a table and then use table.concat to convert it to a string.

Alternately use
file = io.open(strProjectFile,'r')
data = file:read('*all')
file:close()

Which will load the whole file, if you check out PIL you will see that the thing Lua is worst at is concatenating strings.

Post by **tatewise** » 15 Mar 2012 13:37

Thanks Jane, that is useful advice that I will put into practice.

I experimented using that method with the _nameindex.html file not enough memory problem, and it does fix the problem.

Code: Select all

      local strNameIndex = ''
      local tblNameIndex = {}
      local intNameIndex = 0
      for strLine in io.lines(StrFolder..'\_nameindex.html') do
          intNameIndex = intNameIndex + 1
          tblNameIndex[intNameIndex] = strLine
--       strNameIndex = strNameIndex..strLine..'n'
--       if strLine:find('<div class='fhcontent fhpage_Index'>') then break end -- Read HTML file up to first <div class=...> in body
      end
      strNameIndex = table.concat(tblNameIndex,'n')

The first commented-out line is the original code.
The second commented-out line was my new fix.

The problem revolved around VERY big GEDCOM databases.
The not enough memory problem was due to the _nameindex.html file being so large (>1MB) that the Plugin literally ran out of memory, or at least took ages to load it.
Slightly smaller, but still large _nameindex.html files made the Plugin sluggish.
The Plugin only needs some HTML from the , and the menubar at the start of , so it now stops loading when it reaches the contents.
It also now only loads the file once per mapping run, instead of per Individual map, which helps speed it up even more.

I chose to use the _nameindex.html file, because it is the only one certain to exist with a known filename.
The _statistics.html file is optional, and the index.html file might have a different name.

YouTube · Post by **Jane** » 15 Mar 2012 15:16

You don't need to use the index.
Just use table.insert(tablename,string)

Also I have been working with very large gedcom files using load *all or table loading with out issue (One I worked on was 8Mb) and have not had out of memory problems so I would guess it's the concat which is the limiting factor.

Iant · Post by **Iant** » 27 Mar 2012 19:54

Having tried Jane's Life Events mapping plugin & had a problem with the 'dots' on the map, so I installed this one to check it out. The first thing I did was to select a location that didn't plot correctly in Jane's plugin and clicked 'Geocode Plot This Location Entry'. It resulted in the same wrong plot on the map.
The location is Elswick, Northumberland, England which is just west of the centre of Newcastle on Tyne. The plot pointed to somewhere up the coast near Blyth. This seemed strange so I went into Google maps and searched for Elswick, Northumberland, England and got several hits, all bar one in the right area. The odd one was up near Blyth. However this reference was to '203 Elswick Battery Royal Artillery Volunteers' and was the top entry in the list of results.
I tried the same thing with 'Newcastle On Tyne, Northumberland, England' and this too plotted wrongly, and in the same place as in the earlier plugin. Strangely checking directly on Google Maps there are again several hits but the one plotted by the plugin doesn't appear. What is happening?
Ian

Cambiz · Post by **Cambiz** » 27 Mar 2012 21:13

I think Google has Elswick in Tyne and Wear. Use the substitute box and either put just Elswick in or replace Northumberland.

Iant · Post by **Iant** » 27 Mar 2012 21:42

I'm getting round it by getting the locations plotted correctly in Google then getting the coordinates from Get Directions and putting them into the plugin fields.
I suppose this problem is likely to happen when places in the FH Gedcom use the historical counties, but they are today in the modern counties.
Ian

Cambiz · Post by **Cambiz** » 27 Mar 2012 21:54

Another way is to search for (name of place) latitude. Then enter the latitude / longitude into the boxes.

There are a few websites that allow you to get lat/long by clicking on a map - helpful for entries such as 'lost at sea, four miles east of Dover'

I think this plug-in is brilliant and I'm amazed and indebted to Mike for doing it.

Post by **tatewise** » 27 Mar 2012 23:16

Ian says

I suppose this problem is likely to happen when places in the FH Gedcom use the historical counties, but they are today in the modern counties.

That hits the nail on the head, and is why the Plugin offers various ways of adjusting the plot.

You can as you say use the Latitude & Longitude fields.

There are various Substitute values that work well.
Try using the modern address including a local postcode, such as:
Elswick, Newcastle, England
Elswick, UK NE4 7ST

Sometimes a location includes a house or building name, such as Devon House, Bradford that will plot in Devon rather than Bradford.
The solution is to put the house or building name in string quotes 'Devon House', Bradford and it will be excluded from the Geocoding.

Another technique is to use original old place names in the Place field and put modern addresses with postcodes in the Address field.
Then if you use the Plugin option to map Address...Place names it usually does a better job.

See the Help & Advice pages for more advice on options.

Post by **tatewise** » 08 May 2012 12:28

Map Life Facts V2.8 is now available from the Plugin Store with the features listed in this other thread.

See the updated Help & Advice pages listed below for details.

Family Historian User Group