Some experiments at Wayfair can last 60 days or more. To speed up learning in experiments while still optimizing for long term rewards, our team developed a data science platform called Demeter, that uses ML models to forecast longer term KPIs based on customer activity in the short term. In this post we provide an overview for Demeter and its theoretical foundation in causal inference
11 Min Read
Experimentation plays a central role in understanding the business impact of data science strategies and solutions. A few weeks ago, Wayfair Data Science Manager Jerry Chen shared one way Wayfair has improved the experimentation process by building a unified test design and measurement platform (Gemini) for our marketing AB tests (read the blog post here!). In this video, Jerry will provide an introduction to running large scale Monte Carlo simulations to validate/optimize test design and measurement methodology using historical data.
1 Min Read
At Wayfair, data scientists help optimize our marketing channel performance by rapidly and iteratively developing machine learning algorithms and data-driven strategies. When it comes to measuring model efficacy, AB testing plays a central role and serves as a gold standard in strategic decision making within marketing. Due to the natural complexities in marketing operations, marketing-related test design and measurement require special techniques to account for multidimensional, sparse, seasonal, and autocorrelated observations. In this blog post, we will discuss these challenges, and introduce how Wayfair’s advanced marketing test platform —Gemini— remedies these issues.
7 Min Read
This week in Wayfair Data Science’s explainer series, Tim O’Connor discusses experimentation in the context of data science. Experiments are crucial to data science, helping to determine which version of a model is to use in future iterations of a system or generating new sources of data unavailable in a company’s historical data. As such, designing and conducting experiments is a core capability for data scientists at Wayfair. In this video, Tim explains the basics behind A/B tests and synthetic controls, and discusses how they are used to tackle various questions at the company.
1 Min Read
Uplift models seek to predict the incremental value attained in response to a treatment. For example, if we want to know the value of showing an advertisement to someone, typical response models will only tell us that a person is likely to purchase after being given an advertisement, though they may have been likely to purchase already. Uplift models will predict how much more likely they are to purchase after being shown the ad. The most scalable uplift modeling packages to date are theoretically rigorous, but, in practice, they can be prohibitively slow. We have written a Python package, pylift, that implements a transformative method wrapped around scikit-learn to allow for (1) quick implementation of uplift, (2) rigorous uplift evaluation, and (3) an extensible python-based framework for future uplift method implementations.
8 Min Read
At Wayfair, the data science teams collaborate with other partner teams to translate business problems into analytical frameworks, leverage data and machine learning to make robust predictions and recommendations, and build engineering architecture to scale up machine learning solutions. In a recent machine learning seminar at Northeastern University in Boston, Data Science Manager Jen Wang gave an overview of the team’s structure and a few examples of machine learning models used to solve business challenges at Wayfair. The final part of the seminar illustrates a case study (beginning at 23:40 in the video) of how marketing data scientists at Wayfair use uplift modeling to drive incremental revenue in display remarketing.
5 Min Read
As data scientists, we face a variety of problem types. One of our critical challenges is identifying the proper methodological approach to solve each problem. By doing this, we avoid force-fitting the wrong tool to solve a problem, and we avoid having to reformulate a question to fit a specific method.
2 Min Read