Retail Personalization Index Methodology

The purpose of Sailthru's Retail Personalization Index is to quantify how leading retailers are personalizing their customer experience, which personalization methods are most effective, and the relationship between a personalized experience, customer satisfaction, and customer retention.

Research for Sailthru’s 3rd annual Retail Personalization Index study was conducted between June and August 2019.

  1. Scoring retail and ecommerce brands  on how they personalize the customer experience, and overall customer experience across a brand’s website, email program, mobile app
  2. A survey of consumers for each retailer to quantify how those customers felt about the brand and their experiences

Choosing and Scoring the Top 100 Brands on the Index 

Two hundred and fifty brands were selected for inclusion in the study based on a number of criteria, including brand prominence, retailer type, and known leadership in customer experience innovation and personalization. Selected retailers include those with a global presence and those with specific presence in the United States and/or United Kingdom.

Sailthru’s team of personalization experts, including individuals with backgrounds in email marketing, website optimization, mobile messaging, business intelligence, analytics, and broader digital marketing strategies developed a list of criteria on which to measure the selected retailers against. 

The selected criteria, outlined below, included a variety of brand-side and consumer-side activities either in digital channels or using digital data that would be visible to customers shopping with the brand either online or in-store. The criteria list was broken into seven sections or channels: Site, Email, Mobile, Offline, Privacy, Other Digital, and Bonus.

An internal team at Sailthru was assigned to review each retailer in a process called “experiencing the brand,” in which the team would go through the retailer’s site, app, email, product purchase experiences and so on, as if they were a new customer, keeping track of the efficiency of the customer experience and how well their experience was personalized.

The customer experience and personalization criteria against which brands were evaluated include:

SITE

Products

Does the site have on-site product recommendations of any type?

Can you log in and create a profile on the site?

  • Profile: Can you rate, star, favorite, or like products?
  • Profile: Can you create a wishlist or watch list for items? (do not count registries)
  • Does the site allow you to get notifications for out-of-stock or preview items?
  • Does the site allow you to choose whether these notifications are either email or mobile?
  • Is the site front-page content personalized for logged-in users? Does it change after you log in?
  • Does the personalized content have products you’ve looked at before?
  • Ability to view recently-viewed items on dashboard or front page
  • Does the site have modals or overlays with email signup, personalized messaging, product recommendations, or other customzed content?

Content

Is there non-product content, such as a blog, magazine, or non-shop app?

  • Is the content of that blog/app/magazine personalized or curated in some way?

Chatbots

Does the site have a chat feature (either to speak with a human or a bot?)

Offers

  • Are the offers shown on the website consistent with those shown in the mobile app and in emails? (are they for the same products, and the same level of offer?)

Digital Personal Shoppers

Do they have a digital personal shopper, or a really robust product search or recommendation engine? Count anything that makes product recommendations based on criteria, and isn’t just a filtered list.

Preferences

Does the site have a preferences manager?

  • Does the preferences manager allow you to set preferences for products or product recommendations?
  • Does the profile include and store product specifications or preferences – e.g. your clothing size, room sizes, purchased products?
  • Does the preferences manager allow you to set preferences for messaging type/format?
  • Does the preferences manager allow you to set messaging frequency (beyond just “all” or “none”)?
  • Does the preferences manager allow you to opt-out of messaging/tracking? (Do not count opt-outs through email text)

Predictive Technology

Does the site have attribute-based product recommendations? (These are recommendations based on things about the product, so showing you related or similar products, “complete the look”, or products that go with your purchase)

Does the site make audience- or collaborative filtering-based recommendations?

EMAIL

Email Personalization

Do they use your name or username–but not email address as a name–in email messaging?

Do they use collected sign-up/profile data to personalize email content? (common attributes to look for are gender, clothing size, age, lifestyle, location, or whether you have children)

Do the emails they send contain personalized product recommendations? (are they based on your listed preferences or products you’ve viewed or purchased?)

Do they acknowledge your birthday via email, and/or send a gift?

Was one or more of the e-mail subject lines you received personalized?

Welcome Series

Do they send a welcome email after sign-up/registration?

  • Does the welcome email include personalized content (e.g. customer name, products viewed?)
  • Do they send more than one welcome email? (Is there a series to onboard you?)
  • Do they provide clear next steps in their welcome email for further engaging with the brand, such as downloading the app or reading content?

Cart Abandonment

Do they send an email or mobile notification/message for abandoned carts?

  • Do the abandoned cart emails contain one or more of the products left in the cart?
  • Do abandoned cart emails have personalized recommendations for other items?
  • Do abandoned cart emails contain personalized discounts, offers, or messaging around available discounts?
  • Is the channel the abandoned cart message sent in the same as the last channel you engaged in? (if you were last browsing/adding items in-app, was the notification in-app?)

Lifecycle Optimization

Does the timing of emails and messages reflect your activity and tenure as a customer? (Or do they just send you a bunch of regularly-scheduled blasts?)

Are there emails sent based on timing triggers, such as one week after sign-up, post-app download, and so on?

Does the volume of emails received scale with engagements?

Re-engagment/Win-back campaigns

Is there an engagement/email before I disengage? (look for attempts to regain your attention after you have finished the analysis that acknowledge your lack of activity)

Is E-mail volume is reduced if I don’t engage for a period of time?

Purchase-Related

Do post-purchase emails or app messages contain product recommendations based on past orders/similar products?

If a purchase was made, did you receive a follow-up email for purchase satisfaction?

MOBILE

Mobile App

Do they have a mobile app?

Do they have a mobile-enabled website?

Does the app send push notifications?

  • Do push notifications contain personalized recommendations?

Does the mobile app allow users to receive SMS messages?

Does the app have an onboarding experience that asks for product preferences?

Does the app appear to be personalized based on preferences set either in mobile onboarding or on website settings?

Are app push notifications triggered by behavior?

Does the app have rich in-app messaging?

Do they send personalized product recommendations in push notifications/messaging?

Does the app display the customer’s name or username?

Does the app share customer login/profile with the site?

Is the customer’s shopping cart shared across the site and mobile device?

Does the app have ads?

Are mobile notifications or messages triggered by email activity? (We are looking for connections between the two)

Do brand emails contain deep links to the mobile app when clicking through mobile email AND/OR do the emails suppress mobile app download CTAs/messages when you have already downloaded the app?

Mobile app includes either AR or VR features/functionality, or both, to augment product browsing / purchase experience

OTHER DIGITAL

Digital Advertising

Does the retailer retarget customers who abandon carts with display or video advertising? (do the ads mention your abandoned cart explicitly?)

When I browse the internet after adding items to my cart, do retargeted ads contain products that I viewed or added to my cart?

Is there customized messaging in retargeted ads (beyond just personalized products?)

Social

Does the brand have a presence on one or more major social sites? (e.g. Facebook, Twitter, Instagram, Snapchat)

Does the brand use or feature social content from other customers for its recommendations/product images? (looking for where they’re using customer images or social content, so like “see it on other customers” or similar uses)

  • Are the social/user-generated reviews from customers with similar sizes/measurements/preferences to the user? Or are obviously personalized in some way?

Customer Engagement

Does the brand respond to or otherwise interact with customers directly through one or more of these social channels?

OFFLINE

In-Store

Does the brand use the customer’s digital data/profile to personalize or enhance that customer’s experience in-store in any way? (most common: Mobile beacons, but anything they do that’s a novelty works here)

Does the brand site allow the customer to view an item’s in-store availability from the website or app?

Does the brand site allow customers to order items online and pick them up in-store?

Loyalty Programs

Does the brand have a loyalty program? (do not count programs attached to a store credit card)

Does the program have rewards?

  • Do the rewards include access to special sales or in-store/offline events?

Does the brand have a referral program?

Do brand emails contain loyalty program membership information such as total lifetime points and a link to how to use them?

PRIVACY

Does the site have clear messaging when you first arrive telling you that it uses cookies to track your information?

Do you have to explicitly OPT IN to being tracked with cookies? (that is, the message doesn’t just say “hey, we use cookies, click here if you want to change that”)

Does the site make clear and easily understandable how your data will be used?

Newsletter/email subscription boxes are NOT pre-checked when signing-up or checking out

Is opting out is easy to do?

Is unsubscribing is easy to do?

BONUS

Does the brand track a single profile across web, email, and mobile, and somehow connect it to an in-store experience?

The Consumer Survey

To accompany the scoring process, in 2019 Sailthru conducted a survey of 1,500 adults ages 18+ recruited from a panel by Qualtrics. These respondents were in the United States and United Kingdom and reported shopping with one or more of the brands that ranked between 1 and 25 in the 2018 Retail Personalization Index in the prior 12 months. Respondents were asked to describe and rate — on a scale of 1 to 5 — their most recent experience with the brand, as well as how well they felt the brand personalized that experience. Respondents were also asked about their likelihood to recommend the brand to a friend or colleague (to obtain Net Promoter Score) and their likelihood to make another purchase from that retailer in the near future.

Creating the Scoring

Once the brands had been rated as listed above, a series of regression analyses were run using data from the customer survey on NPS, experience ratings, and personalization ratings as the dependent variable and the personalization attributes (scored 1 or 0, as noted above) as the independent variables. From this, Sailthru’s research and insights team was able to identify the relative impact of each on the brand’s customer-given scores and used this magnitude to assign a score to each category. This also informed the design to weight the 4 broader categories unevenly, as the analysis suggested that site personalization efforts had a larger impact than the other categories. The final top 100 list ranks the 250 brands that were analyzed in the study based on this scoring methodology.

See Something Wrong?

If you are a retailer included in the survey, and you believe that we have overlooked some of the criteria above, please contact us: we will investigate the error and update our scoring accordingly.

Want To Be Included Next Year?

If you are a retailer not included in the survey, and you believe that you offer an exemplary or innovative customer experience and use of personalization, please contact us to be considered for the 2020 report.

Want more personalization insights?

Personalization is a long-term strategy. Find out where to start.

Get in Touch