One of the core principles of marketing is that messaging customers about what they want or need is much stronger than messaging them about what they don’t. It’s especially important for Wayfair to identify our customers’ needs, since many of our core products are items that are purchased once over a long period of time, such as a bed, sofa, or dining room set. Being able to identify customers' needs as early as possible allows us to effectively communicate with them, improve the customer experience, and drive growth for the business. However, determining these wants or needs precisely can be difficult since doing so requires a quantitative method of representing a customer’s need state for a product or service, as well as a system to determine the optimal communication given that need state. This is where we at Wayfair Data Science step in.
Varying types of recommender systems, models, and software have tried to tackle the problem of clustering customer needs for decades. Our team previously attempted to implement solutions that represented our product catalog in a hierarchical structure, but ultimately found they were not optimal. In this blog, we’ll talk about Lynx, a novel algorithm we have developed for clustering non-hierarchical relationships through a multi-edge network. Lynx is the foundation of our algorithmic marketing engine that helps us determine what kinds of products we think any given customer needs at a given point in time. Lynx addresses the problem of finding a completely customer-centric and data-driven approach to how we should define customer needs.
Customer Functional Needs
At any given time, a customer may have a diverse set of desires and needs. For example:
- I need a new baseball mitt for the upcoming season
- I need a sofa
- I need a reception hall for my wedding
- I need to do a kitchen renovation with all new appliances
- And many, many more!
As you can see, some needs are clearly things Wayfair can help with (ex: a mattress for you to sleep on) while others are not (such as a baseball mitt).
Identifying if a given customer’s need is something we can help with requires us to determine the set of distinct Functional Needs that can be addressed via the zillions of products in our catalog. We use the term Functional Need to loosely capture the function or functions a given product serves in the eyes of the customer. Although this sounds simple, establishing a customer-centric perspective on the relationship between millions of products and hundreds of customer needs is not a trivial exercise. The first, and likely very obvious, relationship is that many products can serve the same Functional Need (ex: a sofa, sectional, and futon all provide you with ‘something to sit on in your living room’). A second, and perhaps more subtle, complexity is that many of our products can fulfill multiple, mutually exclusive needs. For example, when a customer buys a cute 3-drawer accent chest, she could be using it to fulfill a need for towel storage in the bathroom, clothes storage in the bedroom, or a nice accent piece for her living room.
These complex relationships make it very difficult to find an easy solution that maps our Product Classes to our customer needs. These unique challenges led us to develop Lynx rather than a more traditional hierarchical product category structure.
Product Class Multi-Edge Network
To try to determine the customer needs that Wayfair can fulfill, we started at the Product Class level—groupings of related SKUs curated by our fantastic merchandising team. These include classes such as TV Stands, Desks, and Area Rugs. This gave us a solid foundation of products at the scale of thousands, instead of the millions of individual products or product options that Wayfair provides. From here, we wanted to understand the relationship between Product Classes in terms of the Functional Needs they serve from a customer’s perspective. To do so, we started with the simple idea that each Product Class is related to every other Product Class in a myriad of ways. These relationships can be defined in a multi-edge network with Product Classes as vertices and the edges between them as some measure of relatedness. This is conceptually different from how Product Classes are typically treated by most retailers—a hierarchical system where products sit on a shelf, in a section, in a department at a physical retail store. In our approach, a bench can be in the Outdoor section and the Entryway section—it’s all up to what customers want to use it for!
Data-Driven Functional Needs
Any combination of items that fulfill the same Functional Need are said to be ‘substitutable’ for each other— i.e. they solve the same need in the customer’s mind. Going back to the multi-edge network, we can define a new edge weight that encapsulates substitutability. Intuitively, a set of Product Classes that are highly substitutable will often show strong association in co-views and a weak association in co-purchases—i.e. customers browse many sofas, sectionals, and futons but ultimately only buy one of these. After a lot of testing for various measures for the substitutability edge weights, we settled on a metric that scales positively with the correlation between customer views, and negatively with customer orders.
Functional Needs Algorithm
Once we had developed an edge weight that models the customer behavior that we were trying to capture, we needed to figure out a suitable algorithm to determine which Product Classes are grouped into Functional Needs. This can be modeled as a dense subnetwork identification problem, where each Functional Need is its own dense subnetwork. Detecting these subnetworks is not an easy task—the closest in the literature we were able to find was a hierarchical method. Given the lack of research on dense subnetwork detection in weighted fully connected graphs, we sought to develop a heuristic algorithm that would approximate the solution that we were looking for. The key insight is that we could simplify the problem by only considering subnetworks where each edge was over a certain weight threshold—i.e. including a constraint that all pairwise members of the dense subnetwork pass the threshold.
After reducing the problem, we selected each pair of Product Classes over the chosen threshold, a hyperparameter which was selected through testing. In this testing process, we attempted a range of different thresholds until the Functional Needs developed into a grouping that was intuitively true. For example, we knew the threshold was too low if headboards started joining with mattresses, or too high if inner-spring and foam mattresses started separating into different Functional Needs. Given a queue of the set of candidate Functional Needs, the following algorithm heuristic was applied:
- Select the Functional Need from the front of the queue
- For all Product Classes not already in the Functional Need:
- If the Product Class has an edge over the threshold, add it to the Functional Need and return it to the end of the queue
- If no Product Classes are added, put the Functional Need in the proposed final group
- For all Product Classes not already in the Functional Need:
- Remove Functional Needs that are subsets of other Functional Needs.
- Return remaining Functional Needs
The final output of Lynx produced ~250 dense subnetworks, each containing one or more Product Classes that fulfills the same Functional Need for customers. This provided us with a scalable, customer-centric way of capturing the complex relationship between millions of products and hundreds of customer needs.
The output of this work is used in several ways. We have:
- Incorporated it as part of Data Science models:
- As the target variable for a set of hundreds of propensity models (one model per functional need) that work together to determine what Functional Need a customer is currently looking to fulfill
- Generating model inputs/features based on various customer interactions (clicks, views, add-to-carts, etc.) with each Functional Need (rather than a specific Product Class). These features are inputs into the Functional Need propensity models as well as various other personalization and targeting models
- Shared our learnings with the appropriate functional teams (Storefront and Merchandising) to explore opportunities to improve the customer experience on Wayfair.com (ex: by grouping Product Classes that serve the same Functional Need together, etc.)
- Developed new creative assets and marketing campaigns centered on Functional Needs and Projects rather than individual Product Classes
We expect customer Functional Needs to be relatively invariant over time, but we are always adding new product categories to our selection at Wayfair—so we plan on revisiting and updating Lynx in the future!