Behind the Retail Personalization Index: How We Ranked 100 Brands’ CX


When we began to create our first-ever Personalization Index, we knew it would be a massive undertaking. We already had a methodology framework we were using internally, called Experience the Brand, that our staff uses to better understand brands’ efforts at personalization. Completing an Experience the Brand analysis takes a significant amount of time, but for the Personalization Index, we would have to go much deeper, paying more attention to areas that we traditionally did not investigate because they didn’t align with our suite of solutions.

The first task was simply deciding which brands to include in the study for ranking. There were some brands we knew right away had to be included. Any credible study ranking retailers would have to include Amazon, the ever-expanding juggernaut. Likewise for Nordstrom, a luxury retailer renowned for its customer service. But beyond market leaders, we wanted to include brands known for having more notable personalization strategies, such as JustFab and Sephora. We also wanted a mix of U.S. and U.K retailers. And we wanted a sample of brands that would be representative of a number of niche markets, so we’d have multiple shoe retailers to compare scores for, multiple home goods retailers, and so on.

We also had to decide on the specific criteria we were going to use to evaluate retailers, both within channels and across them. Personalization isn’t just a list of features, it’s about empowering customers to engage with a brand when and how they choose, and making sure those engagements are tailored to that customer in a way that’s relevant to their interests. It was also important for us to acknowledge that while the study focused on the use of digital data and profiles for personalization, the concept of personalization did not begin with the invention of online retail, and we should include traditional channels in our study as well.

While it was tempting to grade brands on how well they executed on certain strategies and features, we ultimately wanted to remain objective in our analysis, and so ended up creating a checklist of questions for each brand to cover how well they personalized the experience. These questions covered the full gamut, from general features, such as whether the brand had personalized product recommendations on site, in emails, and in their mobile app, to more strategic considerations, such as whether the brand scaled the volume of customer emails with engagements.

Pulling away from the crowd

We especially wanted to be on the lookout for things that brands were doing differently – the points of differentiation that made one personalization strategy work better than another. So in our study – and this is also a natural consequence of our regression analysis-based scoring model – retailers scored more points for having features or strategies that were less common. For example, a brand in the study would receive no points for having a Facebook page or a Twitter account because every single brand in our study did that. But not every brand used those social media platforms to communicate with individual customers, which ended up being a significant feature of the NPS models we created, and as a result, was worth a significant number of points in scoring.

Similarly, every brand had some sort of onsite product recommendations. But not every brand used collaborative filtering to generate recommendations based on what other, similar customers are buying. Not every brand had attribute-based recommendations to ensure that appropriate items got in front of customers, nor did every brand use customers’ own exhibited preferences to inform their product recommendations in email communications. These were also major distinguishing factors from a score perspective.

Building sustainable, profitable businesses

Rankings in the Personalization Index also had to be tied to business outcomes. This meant that our scoring had to reflect some “real world” measures of success. Revenue and profit seemed too dependent on other factors, so to define our criteria, we decided to use a mix of customer survey metrics: Primarily Net Promoter Score, but also satisfaction and likelihood to make a future purchase.

To capture this information, we partnered with Qualtrics Panels to conduct a survey of more than 13,000 customers of the 100 brands we studied in the U.S. and the U.K. Survey participants were asked to identify:

  • Their likelihood to recommend the brand to a friend or colleague on a 0 to 10 scale
  • How well they would rate their most recent experience with the brand on a 5-point  scale
  • How well they felt the brand personalized their most recent experience on a 5-point scale
  • Their likelihood to make another purchase from that brand in the next 12 months, on a 5-point scale

We then conducted a series of regression analyses using overall brand scores for each metric as the dependent variable to identify which personalization features and strategies had the greatest correlation with each customer satisfaction score. The results of these analyses were combined and used to create the final scoring rubric for the study.

Opportunities for brands, and for research

Despite the depth of our analysis on the Retail Personalization Index, we couldn’t help noticing areas where next year, we might do things differently. We didn’t really look at voice technologies this year, an area where Amazon currently dominates, but where we might see more traction over the next twelve to twenty four months. We were surprised to find that many brands haven’t made much progress on their mobile strategies, and so there’s a huge opportunity there to potentially expand our research in that direction next year. Most brick-and-mortar stores haven’t really figured out how to use mobile data and technologies to improve the in-store experience. Some are using beacons, but not in particularly creative ways. That’s another big opportunity we want to investigate.

We’ll also be looking at tracking scores year-over-year, seeing who moves up and down, and what strategies pay off from a business standpoint. It’d be naive to claim that personalization is the sole factor in a retailer’s success or failure, but as we broaden the scope of the research and track these results over time, we’ll be able to see how the companies that see personalization as a real business strategy enjoy greater stickiness with customers and succeed in an extremely competitive marketplace.