Power 4.9.2 for our WordPress plugins was released tonight with improved import processing.  Several updates are included as we progress towards improved large location import processing.    All of the upgrades are coming to the MySLP service for our non-WordPress users on services like Wix, Weebly, and Squarespace but we have a bit more testing and integration to ensure 99.999% up time on our SaaS service.

Fixing large location imports

If you were importing thousands of locations and found the import process would get partway through then stop, the Power 4.9.2 patch should help.    If your file could not be processed in one pass by the background import processor, Store Locator Plus would detect the unfinished import and pick up where it left off.  Problem was it would not re-read the header row to match up data to locator fields and no additional locations would come into the location list.    This issue, while leaving the new 4.9 import architecture no better but no worse than the 4.8 and earlier releases, has now been resolved.     The second , and third or fourth passes if necessary, properly pick up where things left off.

The second fix is to prevent that second or third location import pass from starting prematurely when the first pass slows down.    Power 4.9.1 would sometimes think the first import stopped processing and start where the first pass paused.   This slowed EVERYTHING down and if Locations > Import > File Settings > Duplicates Handling was set to Add versus Update it may have duplicated records.

The import issue could manifest for smaller lists of locations if your server is underpowered or your hosting service is severely restricting PHP time slices (processing time) on your account.   A typical moderately-loaded server should be able to easily process 2500 locations within a 60-second PHP time slice with time to spare.     For servers that cannot process imports that quickly this patch will help keep the process going.

Improved import for all list sizes

While traveling the path of fixing the large list imports we also found several ways to improve import performance.  Where we would run through around 600 locations being imported in a 15 second processing time limit ( Settings > General > Server > PHP Time Limit ) with Power 4.9.1 the same file would load all 1,000 locations before the time ran out.     Our 25,000 location test would load nearly 10,000 locations with a typical 60-second PHP time limit.

As an aside, the default time limit setting for SLP is 600 seconds.   We were turning this DOWN to try to force our servers to behave like a severely taxed web server or a severely underpowered server.    Many shared hosting plans will NOT honor this setting and changing it will not have any impact on most server.     MySLP users will not see this setting in Advanced Options as our servers are already tuned to allow maximum PHP processing for large location lists.

Keeping data intact with slashes in CSV files

The most commonly referenced specification for CSV file processing, RFC 4180, dictates what most computer scientist agree is a valid CSV file format.   One part of that specification is how to deal with data that has a quote in it.   Since CSV data that has any special character within is to be wrapped in quotes (“) there has to be a special way to deal with things like a store name My “Special” Store.    RFC 4180 says that the CSV file should use double-quotes to manage this.

Rather than code all of this ourselves we use the built-in PHP CSV file processing functions.    It is far faster than running custom PHP code.

Until our 4.9 releases we had been trying to keep all of our code compatible with PHP 5.2.4 due to the WordPress minimum requirement being 5.2.4 despite their recommendation to use PHP 7.     We finally decided we could not longer support the 11-year old version if we were going to write efficient applications.   Our plugins no longer support PHP 5.2.4 and now require PHP 5.3.

The benefit of this requirement is we can now support the proper RFC 4180 specification instead of the PHP “special Sybase compatible” version that sees the \” character combination as an escape sequence.   This messes up the import with any field that ends with a \ and starts “shifting fields” into the wrong columns.

As of Power 4.9.2 the \ character is treated like any other character which makes the imports a lot happier with files that Excel is mangling.

Power 4.9.2 still requires proper line endings, so for our Mac users running older versions of Excel DO NOT export in Mac compatible mode.   For some reason Microsoft things Mac computers still use the 2003 standard return (\r) line ending instead of the modern standard CRLF (\r\n) line endings that are required for a proper CSV file formatting.    Why Excel allows a CSV export of ANY format that does NOT have a CRLF is a mystery as this is not part of ANY CSV standard including the “special Sybase compatible” version that PHP allows.

Improved import instructions and reporting

One of the obvious changes is some notes are on the Locations > Import tab to help guide you to some of the updates in the 4.9 import process.  This includes explaining the basic 3-step process and the fact that you can find your location CSV file in the Media > Library of WordPress.

The Power 4.9.2 release has simplified the message logging on the bottom of the import page timestamping only the start and end of processing and reporting problem records.   Added, updated, and not updated records are no longer reported on an individual basis in order to improve performance and consume far less memory on large file imports.

In addition to the improved import reporting,  the summary with a count of how many records in the CSV file has added, updated, not updated, were malformed and other summary data is now visible on a per-file basis in the Media Library file details for location CSV files.    This is in addition to the Power 4.9 standard file status , file size, current processing offset, and record count (when completed).

Other tweaks and updates

We snuck in a few smaller things into 4.9.2.

The Identifier column has always had special meaning in CSV files; enabled in General > Data > Add On Data Extensions > Enable Contact Fields, this special column override the sl_id column to match up records to be updated.  While useful for aligning Store Locator Plus with third party data systems it does add some extra overhead to the file processing.    Power will no longer add the overhead to check for that column if the Enable Contact Fields setting is not enabled.  It also means updates will no longer work if you DO include that field in your data source but have not ensured the setting is enabled.   Most sites will not need the Identifier column and update feature.

We fixed the Remote File Retrieval settings.  Somewhere during our updates over the past year these settings stopped saving when you changed them.   We also happened to create a better system for marking “quick save” fields like these the auto-save when changed. You may start seeing quick-save options in more places as they reduce button clicks and are faster than saving entire pages of settings to change one thing.

One last improved import addition — we now report on CSV files that look like they have too many columns.  A CSV file with more than 256 columns is a good indicator that the file is corrupt and the line endings (see Excel and MacOS above) are not set properly.   Rather than create a huge string that includes the entire file and may crash the server, Power 4.9.2 flags this potential error and truncates the logging.   The number of columns (aka fields) is shown in the Media Library details as well and provides more insight into what is going on “behind the scenes” should an import fail.