The Searchspring live indexing API is not intended to fully replace the Searchspring feed bulk indexing. It is only intended to keep product data up to date between bulk indexing runs. Below is a list of best practices when using live indexing to ensure everything runs smoothly.
Keep Your Feed
We recommend that you do not rely entirely upon the Searchspring live indexing API. You should still be using our feed bulk indexing for a few reasons.
- Your feed is an authoritative source of your product data.
- If something goes wrong with your index we can quickly repair any issues by running a bulk index of your feed.
- You should run a bulk index periodically (we recommend nightly) in case there were any updates to your products that were missed.
Use Our Bulk Indexing API
You must use our bulk indexing API to run your bulk indexing on demand. Once your feed generation has started you should pause your live indexing requests (storing them in a queue). Once the feed generation has completed you should use the API to request a bulk index. Once the bulk index is complete you could run your live indexing requests for product updates that have occurred since your feed was generated. You can find documentation for the Bulk Indexing API here.
Limit Number of Live Indexing Requests
There are many times that a product is updated that an update to Searchspring is not required. You should only send Searchspring a live indexing request when data required by Searchspring is changed. Below are some examples:
- If a product goes from having 22 in stock to 21 in stock this does not generally affect Searchspring if you're not displaying quantity in stock in your results display. You should only send a live indexing update if a product goes in or out of stock.
- If a field is updated that isn't being used for Searchspring display, search, sort, or filtering you don't need to send an update.
We support up to 100 concurrent live indexing connections, but we reserve the right to throttle lower in times of high load in order to preserve search performance.
Warning
We recommend limiting the number of products updated in a live indexing request to only include necessary changes. Searchspring will reject live indexing requests that update more than 500 products.
Use Cases & Caveats
At the moment, live indexing should only be used for updating the index in between bulk runs. So just as long as a bulk feed is still the primary way to index data, and a client is providing their own data for things that we usually attach in a supplemental fashion (google analytics, external feeds), on that same feed, then live indexing is able to be used.
Requirements
Full record live indexing “upserts”:
- Always have a bulk feed available for indexing. Can’t be a url to the original feed that never gets updated
- All live indexing request data from client, should be included in next bulk feed
- Can’t have supplemental feeds (ga doc filter; external fields doc filter; any review/rating doc filter). This data would need to already be in the bulk feed
Caveats:
- Live indexing endpoints only acknowledge when data is successfully “queued”. There is no guarantee that the requests will be indexed (even eventually). There is no way to know if a request indexed, and no way to see error messages or reasons should there be an issue.
- Even when requests are successfully indexed, there is no guarantee of order of operation. (exception: newer data for the same record already in queue, will always replace older data)
- If there is some problem on our side, and requests which should have succeeded, but did not because of infrastructure issues, those request are lost. (thus the requirement for a recently updated bulk feed)