A/B testing is one of the most important tools for data-driven product development. It allows teams to estimate the causal impact of a product, design or engineering change by randomly assigning customers to control and treatment experiences. At Wayfair, experimentation runs at scale across the product, and our platform Gambit handles millions of assignments every day. The mechanics are intentionally simple: when a customer loads a page where a test is active, they are immediately bucketed into control or treatment. Assignment is fast, consistent and happens at pageview. But assignment at pageview is not the same as exposure to the treatment.
Consider three tests running simultaneously on a typical product page. The first modifies a carousel of recommendations near the bottom of the homepage. A customer who lands, scans the above-the-fold content and leaves will be assigned to treatment but will never see the carousel. The second test changes the minicart experience. Assignment fires when the page header renders, but the treatment only lands when a customer actually opens the minicart. The third modifies an accordion component. Customers are bucketed the moment the page loads, but the treatment is only visible to a customer who expands it.
In all three cases, the experiment is running correctly. The randomization is valid and the data is clean. Yet, a large share of customers assigned to treatment never encountered the change being tested.
Trigger analysis is Wayfair's approach to this problem. It narrows the reported comparison to the customers who actually had a realistic opportunity to be affected without changing the experiment, the assignment logic or the product.
The Dilution Problem: Why Broad Readouts Lose Signal
The trigger rate of an experiment is the share of assigned customers who actually encountered the treatment. Dilution is what happens when that rate is low. The measured result gets averaged across everyone who was assigned including the large portion who never saw the change. A real effect, present only among the customers who were actually exposed, gets buried in a population it never reached.
As a concrete example, Wayfair recently ran a test that added Artificial Intelligence (AI)-generated review summaries to the product detail page. The feature surfaced a distillation of customer reviews (highlights, common themes and quick sentiment signals) directly on the product detail page. Customers were assigned to the test at pageview, as usual. But the summary only appeared when a customer actually scrolled down below the fold to the reviews section of the page.
The results across the entire population assigned to this test (the all-up results): approximately 4.2 million customers per arm, with an Add-to-Cart Rate lift of −0.01%. Statistically neutral by every standard measure. By this readout, the feature appeared to have had no effect at all. The trigger rate for this test was approximately 34%. Only about one in three assigned customers actually saw the review summary.
The intuition follows directly: if any real effect exists only within the 34% who triggered, the all-up average is diluted by roughly that fraction. A genuine effect among a third of the population, when averaged across the full assigned population, shrinks significantly. Depending on the baseline effect size and the trigger rate, that shrinkage can be enough to push the result from a detectable signal into statistical noise.
This is not an edge case. It is the normal condition for a wide class of experiments at Wayfair.
Table: Example of some experiments most susceptible to dilution at Wayfair
Trigger Analysis and the Math Behind It
Trigger analysis doesn't create a larger effect. It measures the right population. So what does the triggered population actually show? The math and the answer are more revealing than the all-up result suggested.
For the AI-generated review summaries test on the product detail page, when the analysis is restricted to customers who actually saw the review summary (roughly 1.4 million per arm), the picture changes entirely. Add-to-Cart Rate lift: +0.20%, statistically significant. The effect was present throughout the experiment. The all-up readout was simply asking the question at the wrong population. The math explains why.
Let us assume that -
The all-up effect measures the impact of the treatment across everyone who was assigned, whether or not they were actually exposed. Formally, this is known as the Intention-To-Treat (ITT) effect, and it can be decomposed as:
The key assumption in trigger analysis is that customers who never encountered the treatment had no realistic opportunity to be affected by it and so their contribution to the all-up effect is approximately zero. That simplifies the relationship to:
We can verify this directly against our AI review summaries experiment: 34% × 0.20% ≈ 0.07% . Close enough to the near-zero all-up result to confirm the story. The signal was real, but it was diluted across a population two-thirds of which the treatment never reached.
Why does this reduce the sample size you need?
Sample size scales with the inverse square of the effect size: halve the detectable effect, and you need four times as many observations to reach the same statistical power. The implication runs in the other direction too. Recover a larger undiluted effect, and you need far fewer.
In our AI review summaries experiment example, the all-up analysis used approximately 4.2 million customers per arm and found nothing detectable. The triggered analysis used roughly 1.4 million (about one-third as many) and reached statistical significance. The triggered pool was smaller and found the win. For tests with very low trigger rates, the power advantage is larger than it first appears. In user-trigger analysis, all of a triggered user's sessions post first exposure to the treatment are included, not just the one where they triggered. A triggered user who visits multiple times contributes observations from every visit. This matters because statistical power depends on the total number of observations in the analysis, not just the number of users. So a threefold reduction in users does not translate to a threefold reduction in statistical power.
Making Trigger Analysis Work in Practice
Getting trigger analysis right in a production experimentation system requires solving two engineering problems: when to apply the trigger, and how to find a valid counterfactual in the control group. Get either wrong and the result is biased, potentially more misleading than the diluted all-up readout it was meant to replace.
When to apply the trigger: real-time vs post-processing adjustment
There are two places in the experiment lifecycle where a trigger condition can be applied. The first is at assignment: users are only bucketed into the experiment when the trigger fires, so the non-triggered population never enters the test at all. The second after processing experimental results: assignment still happens at pageview as normal, and the trigger condition is applied after the experiment runs, as a filter on the population used to compute results.
We recommend post-processing adjustments and the reasons are practical. Some of them are listed below -
- No changes to the assignment engine. The software development kit (SDK) and pageview bucketing logic remain untouched.
- The full assigned population is preserved throughout the experiment. If a trigger definition turns out to be misconfigured (for example, a "click" event was specified when "view" was the right condition), the experiment does not need to be restarted and the collected data is not lost. The trigger can be corrected and the analysis re-run against the same underlying data. Or the analysis can “fallback” to the diluted pageview assignment logic that’s known to be unbiased.
- No latency at assignment time. Delayed bucketing can create a perceptible lag or a broken user experience. Post triggering avoids this entirely.
[Fig.] Architecture: Trigger Applied Post-Assignment. Assignment occurs at page load (BAU - Business as Usual). The trigger condition is applied later to restrict results to the exposed population only.
One tradeoff of post-processing is data volume. Every user who visits the page gets assigned and tracked through the full pipeline, even those who will never trigger. Only at readout are they filtered out. Real-time triggering avoids this by assigning only users who actually trigger, keeping the experiment population lean from the start. At high experiment volume, that difference in pipeline load adds up.
The real-time alternative is conceptually cleaner: users who never trigger are never assigned, so no post-hoc filtering is needed. In practice, however, it introduces two problems that outweigh that simplicity. First, bucketing a user only after a trigger fires means assignment happens mid-session, after the page has already loaded. That delay creates latency that can degrade the user experience. Second, it requires changes to the assignment SDK that add complexity without changing the statistical outcome. Post-processing adjustments at readout produce the same result with none of that overhead.
Finding the counterfactual control group
Trigger analysis is only valid if the same trigger condition applies equally to both control and treatment. If treatment is filtered down to its triggered users but control is not, the comparison is no longer apples-to-apples. The result will favor treatment regardless of the actual effect.
When the same component exists in both arms, this is straightforward: apply the same tracking event to both. The AI review summaries experiment described in this article fits this pattern as the reviews section of the product detail page was present in both control and treatment, so both arms could be filtered to customers who engaged with it, giving a clean and symmetric comparison.
The harder case is one-sided triggering: treatment introduces a net-new component that does not exist in the control experience. Control customers have no opportunity to trigger, because the component they would trigger on simply is not there. The naive solution to filter treatment to triggered users and leave control unfiltered introduces exactly the bias described above. This problem can be solved using some advanced causal inference methods such as balancing via propensity scores. That approach identifies control users who closely resemble triggered treatment users based on pre-experiment behavior. By reweighting these control users to match the triggered treatment population, a valid comparison can be constructed without needing a symmetric trigger signal in both arms. The tradeoff is additional complexity in validation and a sensitivity to unobserved confounders.
Detecting bias: checking the non-triggered population
One practical way to validate that the zero-effect assumption holds is to run checks on the non-triggered population, the complement of the triggered group. If the assumption is correct, this group should show no treatment effect. A statistically significant result in the complement is a signal that the treatment affected users before any trigger fired, and the triggered estimate should be treated with caution.
Fig: When to Trust a Triggered Readout
Wayfair runs two checks as part of standard triggered readouts. The first is a mean-zero check on the complement population: a test of whether the outcome difference between arms among non-triggered users is statistically indistinguishable from zero. The second is a sample ratio mismatch (SRM) check: a test of whether the trigger fired at equal rates in control and treatment. A systematic imbalance in trigger rates between arms is a sign that the trigger condition itself is correlated with the treatment, which would bias the comparison. Speed metrics are always evaluated on the full assigned population, not just the triggered subset, for the same reason.
Scaling trigger analysis across the enterprise
The operational challenge of running trigger analysis at scale is not only statistics but also consistency. If every test owner defines their own trigger condition from scratch, definitions drift, edge cases get handled differently and the analysis becomes harder to audit and trust.
Wayfair addresses this through a standardized trigger library, which is used to govern trigger definition. This library holds pre-approved, reusable trigger definitions that test owners select from when configuring an experiment. Definitions are centrally maintained, reviewed for correctness and extended as new experiment types emerge. For example, a test owner setting up a below-the-fold carousel test picks the appropriate trigger from our library rather than specifying their own custom event. This reduces the chance of misconfiguration and makes trigger definitions consistent and auditable across tests.
When not to use trigger analysis
Trigger analysis is not always the right tool. Some cases look like good candidates at first glance but have characteristics that make a triggered readout unreliable or unnecessary. Three conditions argue against using it:
- The treatment plausibly affects non-triggered customers. Some changes are broad enough to influence customer behavior before any component-level trigger fires. A global style update, a pricing change visible across the full page or a navigation restructuring are examples. Site Speed is another, for example if a new carousel slows down page load, customers in both arms may behave differently before anyone has scrolled far enough to see it. In these cases the non-triggered population has already been affected, the core assumption is violated and trigger analysis will produce a biased estimate. For speed metrics specifically, the all-up population is always the right scope. For other metrics, the all-up readout is the right fallback.
- The trigger rate is very high. When 80–90% or more of assigned customers trigger the treatment, the all-up readout already captures nearly the full exposed population. The power benefit of trigger analysis might be negligible, and the additional tracking dependency might not be justified by the marginal gain.
- A reliable counterfactual trigger cannot be observed. When there is no way to give the control group a comparable trigger signal, filtering the treatment arm without filtering control introduces the one-sided bias described earlier. If this cannot be resolved, the triggered estimate will be systematically overstated.
Impact and What This Unlocks
Let us return to the experiment that opened this article. The AI review summaries feature was evaluated on 4.2 million assigned customers per arm. The all-up result was −0.01% on Add-to-Cart Rate and not statistically significant. The same experiment, measured on the 1.4 million customers who actually encountered the feature: +0.20% on Add-to-Cart Rate, statistically significant. The feature was not neutral. The all-up readout was. The decision to ship or abandon hung on which population was being asked.
Correct readout: separating signal from noise
Two results look identical in an all-up readout: a feature with no effect, and a feature that never reached the customers it was supposed to. Trigger analysis tells them apart. The first argues for moving on. The second argues for fixing the measurement before drawing any conclusion.
The distinction matters at scale. Below-the-fold components, interaction-gated drawers and targeted experiences are structurally diluted under all-up analysis. The problem is not that the features are ineffective. It is that the question is being asked of the wrong population. Trigger analysis corrects the question. An inconclusive readout becomes an actionable one.
What it cost before vs after trigger testing feature
Before Wayfair built an automated framework around triggered analysis, running a triggered readout was a bespoke manual task. A data scientist spent approximately 80 hours per experiment readout. At Wayfair's scale of thousands of experiments per year, that cost added up fast. This also meant, the teams that could afford the time and had the technical expertise ran it; teams that could not rely on the all-up result, whether or not it was the right question.
Wayfair's automated trigger testing feature we developed applies the trigger condition at read time, computes the triggered and all-up estimates side by side, applies the correct statistical corrections consistently and surfaces the results in the same interface a data scientist uses for any other readout. A triggered analysis that previously took up to 80 hours now takes under 20 minutes. That shift does not merely save time but also changes the baseline expectation: every test on a feature with a defined trigger condition gets a triggered read as a matter of course, not as an optional extra that requires a specialist. Thousands of data science hours are recovered annually and redistributed toward analysis that machines cannot automate.
Beyond time savings, standardization matters too. Manual triggered readouts introduced analyst-to-analyst variability: different people applied corrections at different points in the pipeline, with different assumptions about what counted as a trigger event. The automated framework enforces a single methodology across every readout, removing a quiet source of irreproducibility from the experiment record.
Fewer observations, faster decisions
At a 10% trigger rate, the effect among triggered users is ten times larger than the all-up result. Because sample size scales with the inverse square of the effect size, detecting the triggered effect requires roughly 100 times fewer observations than detecting the diluted all-up effect. The review summaries experiment makes this concrete. The triggered effect (0.20%) was roughly three times larger than the predicted all-up effect (0.07%), requiring about nine times fewer observations to detect. The triggered pool of 1.4 million customers per arm was three times smaller than the 4.2 million in the all-up analysis, but it carried approximately three times more statistical power. The smaller pool found the win.
At trigger rates common to below-the-fold components at Wayfair, typically 10% to 30%, the efficiency gains translate directly to test duration. Tests that would previously have been deprioritized because they were too expensive to measure now have a reliable path to a decision.
Closing Thoughts
Every measurement system encodes assumptions. The most common one in online experimentation is silent: that assignment equals exposure. Trigger analysis corrects that assumption not by changing the experiment, but by asking the right question of the right population. The signal is usually there. The measurement just has to be willing to look in the right place.
