Improved Import With Power 4.9.2

Posted by lcleveland in Map Software, MySLP SaaS, Store Locator, Store Locator Plus, WPSLP

Power 4.9.2 for our WordPress plugins was released tonight with improved import processing. Several updates are included as we progress towards improved large location import processing. All of the upgrades are coming to the MySLP service for our non-WordPress users on services like Wix, Weebly, and Squarespace but we have a bit more testing and integration to ensure 99.999% up time on our SaaS service.

Fixing large location imports

If you were importing thousands of locations and found the import process would get partway through then stop, the Power 4.9.2 patch should help. If your file could not be processed in one pass by the background import processor, Store Locator Plus would detect the unfinished import and pick up where it left off. Problem was it would not re-read the header row to match up data to locator fields and no additional locations would come into the location list. This issue, while leaving the new 4.9 import architecture no better but no worse than the 4.8 and earlier releases, has now been resolved. The second , and third or fourth passes if necessary, properly pick up where things left off.

The second fix is to prevent that second or third location import pass from starting prematurely when the first pass slows down. Power 4.9.1 would sometimes think the first import stopped processing and start where the first pass paused. This slowed EVERYTHING down and if Locations > Import > File Settings > Duplicates Handling was set to Add versus Update it may have duplicated records.

The import issue could manifest for smaller lists of locations if your server is underpowered or your hosting service is severely restricting PHP time slices (processing time) on your account. A typical moderately-loaded server should be able to easily process 2500 locations within a 60-second PHP time slice with time to spare. For servers that cannot process imports that quickly this patch will help keep the process going.

Improved import for all list sizes

While traveling the path of fixing the large list imports we also found several ways to improve import performance. Where we would run through around 600 locations being imported in a 15 second processing time limit ( Settings > General > Server > PHP Time Limit ) with Power 4.9.1 the same file would load all 1,000 locations before the time ran out. Our 25,000 location test would load nearly 10,000 locations with a typical 60-second PHP time limit.

As an aside, the default time limit setting for SLP is 600 seconds. We were turning this DOWN to try to force our servers to behave like a severely taxed web server or a severely underpowered server. Many shared hosting plans will NOT honor this setting and changing it will not have any impact on most server. MySLP users will not see this setting in Advanced Options as our servers are already tuned to allow maximum PHP processing for large location lists.

Keeping data intact with slashes in CSV files

The most commonly referenced specification for CSV file processing, RFC 4180, dictates what most computer scientist agree is a valid CSV file format. One part of that specification is how to deal with data that has a quote in it. Since CSV data that has any special character within is to be wrapped in quotes (“) there has to be a special way to deal with things like a store name My “Special” Store. RFC 4180 says that the CSV file should use double-quotes to manage this.

Rather than code all of this ourselves we use the built-in PHP CSV file processing functions. It is far faster than running custom PHP code.

Until our 4.9 releases we had been trying to keep all of our code compatible with PHP 5.2.4 due to the WordPress minimum requirement being 5.2.4 despite their recommendation to use PHP 7. We finally decided we could not longer support the 11-year old version if we were going to write efficient applications. Our plugins no longer support PHP 5.2.4 and now require PHP 5.3.

The benefit of this requirement is we can now support the proper RFC 4180 specification instead of the PHP “special Sybase compatible” version that sees the \” character combination as an escape sequence. This messes up the import with any field that ends with a \ and starts “shifting fields” into the wrong columns.

As of Power 4.9.2 the \ character is treated like any other character which makes the imports a lot happier with files that Excel is mangling.

Power 4.9.2 still requires proper line endings, so for our Mac users running older versions of Excel DO NOT export in Mac compatible mode. For some reason Microsoft things Mac computers still use the 2003 standard return (\r) line ending instead of the modern standard CRLF (\r\n) line endings that are required for a proper CSV file formatting. Why Excel allows a CSV export of ANY format that does NOT have a CRLF is a mystery as this is not part of ANY CSV standard including the “special Sybase compatible” version that PHP allows.

Improved import instructions and reporting

One of the obvious changes is some notes are on the Locations > Import tab to help guide you to some of the updates in the 4.9 import process. This includes explaining the basic 3-step process and the fact that you can find your location CSV file in the Media > Library of WordPress.

The Power 4.9.2 release has simplified the message logging on the bottom of the import page timestamping only the start and end of processing and reporting problem records. Added, updated, and not updated records are no longer reported on an individual basis in order to improve performance and consume far less memory on large file imports.

In addition to the improved import reporting, the summary with a count of how many records in the CSV file has added, updated, not updated, were malformed and other summary data is now visible on a per-file basis in the Media Library file details for location CSV files. This is in addition to the Power 4.9 standard file status , file size, current processing offset, and record count (when completed).

Other tweaks and updates

We snuck in a few smaller things into 4.9.2.

The Identifier column has always had special meaning in CSV files; enabled in General > Data > Add On Data Extensions > Enable Contact Fields, this special column override the sl_id column to match up records to be updated. While useful for aligning Store Locator Plus with third party data systems it does add some extra overhead to the file processing. Power will no longer add the overhead to check for that column if the Enable Contact Fields setting is not enabled. It also means updates will no longer work if you DO include that field in your data source but have not ensured the setting is enabled. Most sites will not need the Identifier column and update feature.

We fixed the Remote File Retrieval settings. Somewhere during our updates over the past year these settings stopped saving when you changed them. We also happened to create a better system for marking “quick save” fields like these the auto-save when changed. You may start seeing quick-save options in more places as they reduce button clicks and are faster than saving entire pages of settings to change one thing.

One last improved import addition — we now report on CSV files that look like they have too many columns. A CSV file with more than 256 columns is a good indicator that the file is corrupt and the line endings (see Excel and MacOS above) are not set properly. Rather than create a huge string that includes the entire file and may crash the server, Power 4.9.2 flags this potential error and truncates the logging. The number of columns (aka fields) is shown in the Media Library details as well and provides more insight into what is going on “behind the scenes” should an import fail.

Power 4.9.1 Improves Location Import

Posted by lcleveland in MySLP SaaS, Squarespace, Web Presence, Weebly, Wix, WordPress, WPSLP

Store Locator Plus Power 4.9.1 improves location imports. It is available for WordPress now and will be updated in our end-of-year release for MySLP users. Here it what was patched and improved.

All Users (MySLP and WPSLP)

Address 2 column on CSV import files was incorrectly being mapped to the address field. This has been resolved.

The location import processing has vastly improved progress indicators on the import sub tab. Check them out.

Power 4.9.2 Import Processing Progress Bar

WordPress Specific

Store Pages custom post types , that are included with Power, correctly registers with WordPress 4.9 showing the sidebar menu.

Power has been updated to work on servers running PHP 5.3. You really should upgrade to PHP 5.6 if you value site security, not too mention performance. PHP 5.3 has been out-of-date for several years now. Actually jump right to PHP 7 if you can.

General Notes

Import and Basic Auth

The new import process requires that your site be able to run WP Cron. Some hosts and site admins turn that off for “security reasons”, but there is limited-if-any security risk with that process. Not running it not only will prevent the new imports from running but stops lots of other things from working in WordPress; like automated updates.

If your site is using basic authentication, you’ll know if it is, the import processor cannot start the second (reading the CSV file) and third stages (geocoding locations) automatically. You’ll need to be logged in as an authorized user and surf to your website/wp-cron.php to kick things off. Some development and staging sites use basic authentication to prevent a casual web surfer or search bot from accessing a site they are working on.

Marker Not Found Notice On Settings Page

If you are seeing “marker not found” notifications when going to your Settings tab in SLP, this could be an obvious indication that basic auth is running on your site. Our marker-testing uses the built-in WordPress “get this web thing” function which access something, like your marker URL , as if it is a “Joe Public” web user just surfing the web WITHOUT typing in special access codes or passwords. As a side note, if you are seeing that notification there is a good chance that someone visiting your site with a browser you’ve not tested yet is getting a map with no markers on it.

Tag Based Location Searches

No, we’ve not completely fixed the TAG search in this release. The tag searching is better, but still not 100%. We are actively chasing down that bug and a few others for the SLP 4.9.2 + Power 4.9.2 release.

Other Updates

We are working on getting Premier to run on PHP 5.3 servers (you ARE going to upgrade to PHP 5.6, right?) along with some other minor patches.

If you have issues please post in the forum (this shows up in our dev teams Slack channel) or Contact Us via email. If using the forum, please start your own thread if you are sharing information about your site and want help debugging. Include a link to your locator page and copy the content of your plugin environment (for WPSLP users) or profile (MySLP users).

Keep in mind that if you updated OTHER THINGS like WordPress, another plugin, or a theme… those can break Store Locator Plus. SLP is the most visible thing you’ll see if your JavaScript breaks as the search form stops working. Nobody usually notices a slide-out menu not sliding out but a search for locations is front-and-center. Check your browser’s developer tools and look in the console for error messages. We’ve had several sites update things like their them which was missing a library, such as Lightbox, that breaks JavaScript and stops the locator from running.

Location Import Process Improved With Power 4.9

Posted by lcleveland in Map Software, MySLP SaaS, Store Locator Plus, WPSLP

With the release of the WordPress plugins version 4.9 of Store Locator Plus and the Power add on we focused on improving the location import process for large lists. MySLP Power users will see this release by the end of the year after it has passed full testing on our SaaS platform.

Why Did We Change The Process?

Prior to version 4.9 there were a number of limitations that could prevent larger lists of locations from importing properly. For the fastest servers with lots of memory and super-fast networking as well as a premium tier on the Google API service, Store Locator Plus could import 40,000 locations without any problem.

As WordPress started adding more network overhead, all those update notifications and the news feed in the WordPress dashboard comes from somewhere, the network started to become an issue. On top of that Google started tracking every single map request so they could start charging people using their maps on a per-lookup basis. Both of these changes increased the network traffic, slowing down how quickly Store Locator Plus could get a latitude/longitude for those new locations being imported.

Often your server would give up, saying “you had your 30 seconds of my time” and stopping the import mid-stream. The PHP time limitation on processing is often the first thing that “gives out” when importing large lists of locations. For most shared hosts the limit is 30 seconds. Even on a dedicated host it is good practice to not let a PHP process consume more than 120 seconds of CPU time. This prevents “runaway” processes that take over the entire server which can grind EVERYTHING to a halt forcing a server restart.

The bad news is that if you are importing 10,000 locations and the process gives up after 9,800 locations you have to start over. At least that is how it USED TO BE with SLP 4.8 and earlier.

How Imports Improved In Power 4.9

With version 4.8 the location import process was “tied to the browser” until it was complete. When you clicked the “upload CSV” button your browser would sit there attached to the website while the file was uploaded, read line-by-line, each location sent to Google to get a latitude and longitude and then get added to the Store Locator Plus locations list. If the network was slow this could take a long time, often 10 minutes or more. That is FOREVER to keep a browser window open these days.

With Power 4.9 we split this process into 3 stages.

Stage 1 : Get The File On The server

This is the only step of the process that stays “attached to your browser”. The browser will bar processing the page while the file is copied from your laptop to the web server. This is typically a quick process even for a file with 250,000 locations. The time it takes depends on how fast your local network and ISP connection is.

Previous versions stored the uploaded location files in a “hidden” SLP-specific directory under WordPress uploads. Version 4.9 stores them in the media library. This gives you an easy way to see that the file was uploaded properly, see what the current import state is, and later download or delete the files when the import is complete.

Stage 2 : Import The Locations Into SLP

After the file is uploaded a detached process starts to work on the actual location import. This ensures your browser doesn’t time out as this part happens in the background. Your browser is “free to go” while your server does the work of loading in the locations.

It runs a WP Cron job in the background to import the locations WITHOUT geocoding. This is much faster and ensures the import doesn’t disconnect partway through.

As part of improving visibility into this process, Power 4.9.1 will be out soon and will be included in our end-of-year MySLP update. It includes a cool interactive progress bar as well showing you how many records have been read.

If your server times out before our background location loader finishes reading the file , SLP remembers the last record read and re-spawns a new background job to pick up where it left off. It will do this until it reaches the end of the file no matter how often PHP times out.

Stage 3: Geocode The Locations

AFTER the file has been loaded it will then geocode all uncoded locations. That means you may catch SLP with a list of locations none of which are geocoded and not showing up on the map. This is a temporary state while SLP talks to Google. Since that is always the slowest part of an import we decided to do this in a separate process so at least SOME locations start showing up right away and eventually of them are loaded.

This is similar to the import. It does it in the background and if PHP times out it will re-spawn itself until it has tried to geocode very location at least once. This is a big step toward not having an import time out. It also keeps you from manually having to re-geocode if you hit your daily 2500 location limit and didn’t opt for a paid API key.