Improved Import With Power 4.9.2

Posted by lcleveland in Map Software, MySLP SaaS, Store Locator, Store Locator Plus, WPSLP

Power 4.9.2 for our WordPress plugins was released tonight with improved import processing. Several updates are included as we progress towards improved large location import processing. All of the upgrades are coming to the MySLP service for our non-WordPress users on services like Wix, Weebly, and Squarespace but we have a bit more testing and integration to ensure 99.999% up time on our SaaS service.

Fixing large location imports

If you were importing thousands of locations and found the import process would get partway through then stop, the Power 4.9.2 patch should help. If your file could not be processed in one pass by the background import processor, Store Locator Plus would detect the unfinished import and pick up where it left off. Problem was it would not re-read the header row to match up data to locator fields and no additional locations would come into the location list. This issue, while leaving the new 4.9 import architecture no better but no worse than the 4.8 and earlier releases, has now been resolved. The second , and third or fourth passes if necessary, properly pick up where things left off.

The second fix is to prevent that second or third location import pass from starting prematurely when the first pass slows down. Power 4.9.1 would sometimes think the first import stopped processing and start where the first pass paused. This slowed EVERYTHING down and if Locations > Import > File Settings > Duplicates Handling was set to Add versus Update it may have duplicated records.

The import issue could manifest for smaller lists of locations if your server is underpowered or your hosting service is severely restricting PHP time slices (processing time) on your account. A typical moderately-loaded server should be able to easily process 2500 locations within a 60-second PHP time slice with time to spare. For servers that cannot process imports that quickly this patch will help keep the process going.

Improved import for all list sizes

While traveling the path of fixing the large list imports we also found several ways to improve import performance. Where we would run through around 600 locations being imported in a 15 second processing time limit ( Settings > General > Server > PHP Time Limit ) with Power 4.9.1 the same file would load all 1,000 locations before the time ran out. Our 25,000 location test would load nearly 10,000 locations with a typical 60-second PHP time limit.

As an aside, the default time limit setting for SLP is 600 seconds. We were turning this DOWN to try to force our servers to behave like a severely taxed web server or a severely underpowered server. Many shared hosting plans will NOT honor this setting and changing it will not have any impact on most server. MySLP users will not see this setting in Advanced Options as our servers are already tuned to allow maximum PHP processing for large location lists.

Keeping data intact with slashes in CSV files

The most commonly referenced specification for CSV file processing, RFC 4180, dictates what most computer scientist agree is a valid CSV file format. One part of that specification is how to deal with data that has a quote in it. Since CSV data that has any special character within is to be wrapped in quotes (“) there has to be a special way to deal with things like a store name My “Special” Store. RFC 4180 says that the CSV file should use double-quotes to manage this.

Rather than code all of this ourselves we use the built-in PHP CSV file processing functions. It is far faster than running custom PHP code.

Until our 4.9 releases we had been trying to keep all of our code compatible with PHP 5.2.4 due to the WordPress minimum requirement being 5.2.4 despite their recommendation to use PHP 7. We finally decided we could not longer support the 11-year old version if we were going to write efficient applications. Our plugins no longer support PHP 5.2.4 and now require PHP 5.3.

The benefit of this requirement is we can now support the proper RFC 4180 specification instead of the PHP “special Sybase compatible” version that sees the \” character combination as an escape sequence. This messes up the import with any field that ends with a \ and starts “shifting fields” into the wrong columns.

As of Power 4.9.2 the \ character is treated like any other character which makes the imports a lot happier with files that Excel is mangling.

Power 4.9.2 still requires proper line endings, so for our Mac users running older versions of Excel DO NOT export in Mac compatible mode. For some reason Microsoft things Mac computers still use the 2003 standard return (\r) line ending instead of the modern standard CRLF (\r\n) line endings that are required for a proper CSV file formatting. Why Excel allows a CSV export of ANY format that does NOT have a CRLF is a mystery as this is not part of ANY CSV standard including the “special Sybase compatible” version that PHP allows.

Improved import instructions and reporting

One of the obvious changes is some notes are on the Locations > Import tab to help guide you to some of the updates in the 4.9 import process. This includes explaining the basic 3-step process and the fact that you can find your location CSV file in the Media > Library of WordPress.

The Power 4.9.2 release has simplified the message logging on the bottom of the import page timestamping only the start and end of processing and reporting problem records. Added, updated, and not updated records are no longer reported on an individual basis in order to improve performance and consume far less memory on large file imports.

In addition to the improved import reporting, the summary with a count of how many records in the CSV file has added, updated, not updated, were malformed and other summary data is now visible on a per-file basis in the Media Library file details for location CSV files. This is in addition to the Power 4.9 standard file status , file size, current processing offset, and record count (when completed).

Other tweaks and updates

We snuck in a few smaller things into 4.9.2.

The Identifier column has always had special meaning in CSV files; enabled in General > Data > Add On Data Extensions > Enable Contact Fields, this special column override the sl_id column to match up records to be updated. While useful for aligning Store Locator Plus with third party data systems it does add some extra overhead to the file processing. Power will no longer add the overhead to check for that column if the Enable Contact Fields setting is not enabled. It also means updates will no longer work if you DO include that field in your data source but have not ensured the setting is enabled. Most sites will not need the Identifier column and update feature.

We fixed the Remote File Retrieval settings. Somewhere during our updates over the past year these settings stopped saving when you changed them. We also happened to create a better system for marking “quick save” fields like these the auto-save when changed. You may start seeing quick-save options in more places as they reduce button clicks and are faster than saving entire pages of settings to change one thing.

One last improved import addition — we now report on CSV files that look like they have too many columns. A CSV file with more than 256 columns is a good indicator that the file is corrupt and the line endings (see Excel and MacOS above) are not set properly. Rather than create a huge string that includes the entire file and may crash the server, Power 4.9.2 flags this potential error and truncates the logging. The number of columns (aka fields) is shown in the Media Library details as well and provides more insight into what is going on “behind the scenes” should an import fail.

Location Import Process Improved With Power 4.9

Posted by lcleveland in Map Software, MySLP SaaS, Store Locator Plus, WPSLP

With the release of the WordPress plugins version 4.9 of Store Locator Plus and the Power add on we focused on improving the location import process for large lists. MySLP Power users will see this release by the end of the year after it has passed full testing on our SaaS platform.

Why Did We Change The Process?

Prior to version 4.9 there were a number of limitations that could prevent larger lists of locations from importing properly. For the fastest servers with lots of memory and super-fast networking as well as a premium tier on the Google API service, Store Locator Plus could import 40,000 locations without any problem.

As WordPress started adding more network overhead, all those update notifications and the news feed in the WordPress dashboard comes from somewhere, the network started to become an issue. On top of that Google started tracking every single map request so they could start charging people using their maps on a per-lookup basis. Both of these changes increased the network traffic, slowing down how quickly Store Locator Plus could get a latitude/longitude for those new locations being imported.

Often your server would give up, saying “you had your 30 seconds of my time” and stopping the import mid-stream. The PHP time limitation on processing is often the first thing that “gives out” when importing large lists of locations. For most shared hosts the limit is 30 seconds. Even on a dedicated host it is good practice to not let a PHP process consume more than 120 seconds of CPU time. This prevents “runaway” processes that take over the entire server which can grind EVERYTHING to a halt forcing a server restart.

The bad news is that if you are importing 10,000 locations and the process gives up after 9,800 locations you have to start over. At least that is how it USED TO BE with SLP 4.8 and earlier.

How Imports Improved In Power 4.9

With version 4.8 the location import process was “tied to the browser” until it was complete. When you clicked the “upload CSV” button your browser would sit there attached to the website while the file was uploaded, read line-by-line, each location sent to Google to get a latitude and longitude and then get added to the Store Locator Plus locations list. If the network was slow this could take a long time, often 10 minutes or more. That is FOREVER to keep a browser window open these days.

With Power 4.9 we split this process into 3 stages.

Stage 1 : Get The File On The server

This is the only step of the process that stays “attached to your browser”. The browser will bar processing the page while the file is copied from your laptop to the web server. This is typically a quick process even for a file with 250,000 locations. The time it takes depends on how fast your local network and ISP connection is.

Previous versions stored the uploaded location files in a “hidden” SLP-specific directory under WordPress uploads. Version 4.9 stores them in the media library. This gives you an easy way to see that the file was uploaded properly, see what the current import state is, and later download or delete the files when the import is complete.

Stage 2 : Import The Locations Into SLP

After the file is uploaded a detached process starts to work on the actual location import. This ensures your browser doesn’t time out as this part happens in the background. Your browser is “free to go” while your server does the work of loading in the locations.

It runs a WP Cron job in the background to import the locations WITHOUT geocoding. This is much faster and ensures the import doesn’t disconnect partway through.

As part of improving visibility into this process, Power 4.9.1 will be out soon and will be included in our end-of-year MySLP update. It includes a cool interactive progress bar as well showing you how many records have been read.

Power 4.9.2 Import Processing Progress Bar

If your server times out before our background location loader finishes reading the file , SLP remembers the last record read and re-spawns a new background job to pick up where it left off. It will do this until it reaches the end of the file no matter how often PHP times out.

Stage 3: Geocode The Locations

AFTER the file has been loaded it will then geocode all uncoded locations. That means you may catch SLP with a list of locations none of which are geocoded and not showing up on the map. This is a temporary state while SLP talks to Google. Since that is always the slowest part of an import we decided to do this in a separate process so at least SOME locations start showing up right away and eventually of them are loaded.

This is similar to the import. It does it in the background and if PHP times out it will re-spawn itself until it has tried to geocode very location at least once. This is a big step toward not having an import time out. It also keeps you from manually having to re-geocode if you hit your daily 2500 location limit and didn’t opt for a paid API key.

WordPress Store Locations SaaS

Posted by lcleveland in Map Software, MySLP SaaS, Store Locator, Store Locator Plus, WPSLP

Some of our long-time WordPress plugin customers have decide to reduce their workload and customer maintenance overhead by migrating to the MySLP service. Here are your answers to some of the most common questions that have come up during the process.

How Do I Connect MySLP To The Plugin?

The short answer: You don’t.

The MySLP service is a self-contained application that installs on your WordPress site , or any site builder that supports JavaScript, with a simple JavaScript embed. It does not require the Store Locator Plus plugins to be installed on the WordPress site.

In fact, if you are now using MySLP you should delete any of the Store Locator Plus plugins you have installed on your WordPress site. Make sure you’ve downloaded your locations and copied any settings you wish to take to MySLP beforehand. While on that topic, a good site administrator will delete any obsolte and unused plugins from a WordPress site. Deactivated plugins are still available for hackers to exploit.

If There Is No Plugin How Do I Use It?

To manage the locations and settings you will login to your MySLP Dashboard.

When you are ready to deploy the map you can go to any page or post on the site and paste the embed code you copied from your MySLP dashboard. Make sure you are viewing the page/post editor in text mode first, then paste the JavaScript embed where you want the map to appear on the page.

How Do I Get The Locations Into MySLP?

If you are migrating an existing site you will want to export locations from the WordPress install with the Power add on BEFORE deactivating and deleting the WPSLP plugins.

Go to the Locations tab in Store Locator Plus and select “Export, Hosted” or “Export, Download” from the download menu then click the To All portion of the Apply To All button.

Make sure you’ve copied your settings to your new MySLP account as well.

Once you have your locations CSV file go to your MySLP Dashboard account and look for the Locations Import tab under Advanced Options. Import is available on Power or higher level accounts. You can upgrade or downgrade service to/from the Advanced level as needed to get the import feature during the migration process.

On a related note, we are working on an “Import From A WPSLP Install” feature for MySLP that will bring over locations directly without the manual import/export process.

How Many Sites Can Use My Map?

Your MySLP account can deploy the locator on as many sites and mobile apps as you need. Each deployment will share the same map settings and location list which can be useful for building SEO content pages while retaining a consistent location listing between the sites and pages.