Big data: Personalizing the customer experience in real time

Delivering a personalized customer experience has been a retail mantra for decades, but in 2014 it is taking on new dimensions to address the tectonic shifts in consumer buying patterns.

Retailers were rocked by a disappointing Black Friday even as Cyber Monday 2013 showed record results, with consumers spending more than $2 billion, and mobile purchases that day increased 55 percent over the previous year. Moreover, a 2013 study by comScore, "Marketing to the Multi-Platform Majority," reported that more than half of digital consumers in the U.S. already engage online via both computers and mobile devices.

With more consumers shopping online and via mobile devices, retailers now face a clear mandate to personalize their interactions on the fly based on up to-the-moment data about what buyers are looking at, where they are, and what devices they're using. Already some retailers are leading the charge by implementing systems to effectively combine historical and real-time data in their user profiles to engage consumers in a meaningful way — whether they're browsing online or wandering the store aisles.

Significantly, pioneering retailers are not throwing out old best practices. Instead, they are capitalizing on new technologies to engage in more nuanced segmentation while responding in real time. With this in mind, we will look at the data management systems required for real-time personalized consumer interactions, how to apply the Pareto 80/20 rule to capitalize on clickstream data in addition to traditional data sources, and potential pitfalls to avoid when engaging in real-time online interactions.

Expanding the 360-degree view

Story continues below...

A Global Retail Custom Fixture Company
Sparks is a full-service retail custom fixture company providing a variety of integrated services for brands both large and small. We create and produce innovative ideas that deliver continued value across every type of retail environment.  

Until recently, a retail application's personalization of offers was largely based on historical data stored within a data warehouse and more recently an Apache Hadoop cluster for backend analysis. Added to the mix was information from operational systems, such as inventory, pricing, payments and logistics, using structured data stored in one or more relational databases.

However, relational databases are not designed to capture the range of unstructured data coming in from the Web and mobile devices, such as clickstreams, page views and geographical data. As a result, retailers are starting to add not-only-SQL (NoSQL) databases or in-memory databases, which can process large volumes of both structured and unstructured data at extremely high speed, often returning responses in one-one-hundredth to one-tenth of a second.

In this new scenario, retail applications running on one or more application servers feed all user activity and context data, structured and unstructured, into the data warehouse and/or Hadoop cluster. As part of the analysis, users are tagged or categorized into segments, and this segment data is periodically moved to augment user profiles stored in the in-memory database. The application servers also write clickstreams and current user activity date directly into the in-memory database, mapping session IDs, cookies, device IDs, IP addresses, and other user identities across the entire spectrum of platforms and channels.

With this deployment, when a user clicks, swipes or otherwise interacts with an application, the application server draws on the in-memory database for immediate access to a view of the customer that combines the segmentation data with at-the-moment information about what the consumer is doing and where.

Applying the 80/20 rule to real-time big data

With systems in place to analyze unstructured data, retailers can now experience an uplift of 18 percent to 22 percent by doing very simple, straightforward behavioral profiling based on clickstreams.

One early project with a major retailer's Internet site started off by employing the Pareto 80/20 rule — using broad customer segments from clickstream data, A/B testing, and variance testing to create about 25 different profiles each for abandonment and for conversion. Additionally, conducting a very rigorous, structured regression formula against those 25 data points helped to find emerging patterns. Spotting these patterns made it possible to funnel ads and data into the experience of a particular user, to minimize the abandonment behaviors and maximize the conversion behaviors, creating a 21 percent uplift.

Enabling technologies for managing big data make it possible to create very fine segments. The 80/20 rule still applies, but instead of one or two dozen fairly broad customer segmentations, some retailers are realizing that they can easily have 100 or more customer segmentations on which to apply their algorithms for relevant offers.

As soon as a behavior based on a clickstream is identified, a retailer can then put up ads and other content, as well as personalize interactions and offers for consumers. Significantly, work streams change all the time, so it is important to continually re-evaluate what the flow of content on a page is going to be based on recognized behaviors there. That is the next level of variance in A/B testing.

But is it the right offer?

Consumers comparison-shopping on the Web or their mobile devices are putting new competitive pressures on pricing, but simply offering the lowest price is counter-productive. Consider the survey findings of the National Retail Federation: While 2 million more people shopped in stores and online from Thanksgiving through Cyber Monday in 2013 than in the previous year, the average consumer's spending dropped 3.9 percent to $402.02, and overall purchases fell 2.9 percent to $57.4 billion.

Too often dynamic pricing is a race to the bottom unless the consumer is getting the feeling from the vendor that they are recognized and unique and valued. Instead, personalized offers should look at how to build the overall value of the customer. For instance, knowledge of the customer's previous tall selections combined with real-time data about a search for a shirt could lead an offer of, "We have the tall version of that shirt, which we can ship out today or have available for pick-up at our store." Other options might be a color exclusive to the store or 50 percent off a second shirt of the same style, thereby offering a discount while increasing the total sale.

In short, it is about customization, understanding who is doing the shopping right here, right now and giving them maximum value for their time.

Source: http://goo.gl/63XnUM