Behind the Retail Personalization Index: How We Ranked 100 Brands’ CX
When we began to create our first-ever Personalization Index, we knew it would be a massive undertaking. We already had a methodology framework we were using internally, called Experience the Brand, that our staff uses to better understand brands’ efforts at personalization. We spent all summer experiencing brands, which we selected based on reputation, category and brand prominence.
Any credible study ranking retailers would have to include Amazon, although plenty of retailers have out-personalized the ecommerce juggernaut. Likewise for Nordstrom, a luxury retailer renowned for its customer service. But beyond market leaders, we wanted to include brands known for having more notable personalization strategies, such as JustFab and Sephora. We also wanted a mix of U.S. and U.K retailers. And we wanted a sample of brands that would be representative of a number of niche markets. For example, multiple shoe retailers to compare scores for.
Last year, we had decided on the specific criteria we were going to use to evaluate retailers, both within channels and across them. Personalization isn’t just a list of features. It’s about empowering customers to engage with a brand when and how they choose. It’s also about making sure those engagements are tailored to that customer in a relevant way.
This year, we went much more granular. After more than 500 brands requested their scores the first time around, we decided to ramp up our research. We evaluated more than 250 brands, though we only scored the top 100 brands. This provided us a better understanding of how these retailers are performing, giving them better inspiration on how to approach their approaches to customer experience.
It was tempting to grade brands on how well they executed on certain strategies and features. However, we ultimately wanted to remain objective in our analysis. As a result, we created a checklist of questions for each brand to cover how well they personalized the experience.
These questions covered the full gamut, from general features, such as whether the brand had personalized product recommendations on site, in emails, and in their mobile app, to more strategic considerations, such as whether the brand scaled the volume of customer emails with engagements. Since personalization did not begin with the invention of online retail, we included traditional channels in our study as well.
Pulling Away From the Crowd
We especially wanted to be on the lookout for things that brands were doing differently – the points of differentiation that made one personalization strategy work better than another. So in our study — and this is also a natural consequence of our regression analysis-based scoring model — retailers scored more points for having features or strategies that were less common. For example, brands received no points for having a Facebook page or a Twitter account. Every single brand in our study has one, but they don’t all use them to communicate with individual customers. That turned out to be a significant feature of the NPS models we created. As a result, it was ultimately worth a significant number of points in scoring.
Similarly, every brand had some sort of onsite product recommendations. But not every brand used collaborative filtering to generate recommendations based on what other, similar customers are buying. Not every brand had attribute-based recommendations to ensure that appropriate items got in front of customers. Brands also didn’t all use customers’ own exhibited preferences to inform their product recommendations in email communications. These were also major distinguishing factors from a score perspective.
Building Sustainable, Profitable Businesses
Rankings in the Personalization Index also had to be tied to business outcomes. This meant that our scoring had to reflect some “real world” measures of success. Revenue and profit seemed too dependent on other factors. To define our criteria, we used a mix of customer survey metrics beyond Net Promoter Score.
Partnered with Qualtrics Panels before the inaugural launch, we surveyed more than 13,000 customers of the 100 brands we studied in the U.S. and U.K. We asked respondents about:
- Their likelihood to recommend the brand to a friend or colleague on a 0 to 10 scale
- How well they would rate their most recent experience with the brand on a 5-point scale
- How well they felt the brand personalized their most recent experience on a 5-point scale
- Their likelihood to make another purchase from that brand in the next 12 months, on a 5-point scale
We then conducted a series of regression analyses using overall brand scores for each metric as the dependent variable to identify which personalization features and strategies had the greatest correlation with each customer satisfaction score. We combined the results of these analyses to create the Personalization Index’s final scoring rubric.
Opportunities for Brands and for Research
Despite the depth of our analysis, we couldn’t help thinking about the next Personalization Index. We didn’t really look at voice technologies this year, an area where Amazon currently dominates, but where we will likely see more traction in the near future.
We were surprised at brands’ relatively weak mobile strategies. There’s a huge opportunity there to potentially expand our research in that direction next year. Most brick-and-mortar stores haven’t really figured out how to use mobile data and technologies to improve the in-store experience. Some are using beacons, but not in particularly creative ways. That’s another big opportunity we want to investigate.
We’ll also be looking at tracking scores year-over-year. Who moves up and down? What strategies pay off from a business standpoint? It’d be naive to claim that personalization is the sole factor in a retailer’s success or failure, but as we broaden the scope of the research and track these results over time, we’ll be able to see how the companies that see personalization as a real business strategy enjoy greater stickiness with customers and succeed in an extremely competitive marketplace.