Impact section on the project home
From a result to a daily number
Each shipped change contributes a projected daily impact: the per-user effect it measured, converted to metric units per day, scaled to a full rollout.- The per-user effect is the absolute difference between the shipped variant’s mean and control’s, exactly as the experiment measured it.
- Treatment users ÷ window days turns that into units per day: how many users per day actually saw the winning variant during the analysis window.
- Dividing by the treatment share — the shipped variant’s share of eligible traffic, as the unrounded product of the layer’s traffic slice and the variant split — scales the observed daily total up to what a 100% rollout of that experience would produce.
Intervals travel with the estimate
A point estimate alone would overstate how much you know, so the experiment’s uncertainty is carried through the projection. The impact interval is reconstructed from the measured relative confidence interval on the lift: multiplying it by the control mean recovers the absolute-difference interval, and scaling by the same rollout factor as the point estimate gives the interval on projected daily impact. Because the relative interval is what the analysis reports in both modes, this works identically for fixed-horizon and sequential (anytime-valid) results — a sequential result’s wider interval flows straight into a wider impact interval, as it should.The false-positive discount
Here is the part most impact reports skip. At a 5% significance level, some of your winners are noise — and the fewer of your ideas that truly work, the larger the share of your “wins” that are false. Summing undiscounted winners bakes that winner’s bias into the headline. Traffical discounts each concluded win by its estimated false-positive risk — the probability that a result this significant came from a change with no real effect:- Your observed win rate over concluded experiments can’t be used directly: it estimates the probability of a significant result — a mixture of lucky nulls, arising at rate α, and real detections, at rate 1−β — not π itself. Traffical inverts that mixture to recover π from the win rate.
- Until you have at least 20 concluded experiments, a conservative cold-start prior of π = 0.33 applies — roughly one idea in three moves its metric, the industry’s long-run experience.
- The inferred prior is floored at 0.05, so one unlucky stretch never zeroes out the whole program.
The current prior is disclosed right in the Impact section: the always-visible disclaimer line shows the π in effect, and clicking it opens the methodology popover with the formulas and caveats applied to the numbers you’re looking at.
Group totals
Within a metric, contributions are summed after discounting — but the group’s interval is not the sum of the individual bounds. Summing bounds assumes every experiment erred in the same direction at once, the perfect-correlation worst case, and produces intervals too wide to act on. Instead the group interval treats contributions as independent: the total is the sum of discounted impacts, and its half-width is the root-sum-of-squares of the individual (discounted) half-widths. Independent errors partially cancel, so the combined interval is tighter than the worst case — and the independence assumption is labeled as such in the limitations.
Per-metric impact group with contributions and interval
Window accounting
The Impact view is scoped to a date range (the last 90 days by default, filterable by surface). A change’s contribution to the window total accrues by overlap days:- Shipped before the window starts → it accrues its daily impact for the whole window.
- Shipped mid-window → it accrues only from its ship date, including fractional days.
- No overlap → it contributes nothing.
Shown separately, never added
Two kinds of activity appear in the Impact section but are never summed into the concluded total:- In-flight experiments — live readings of running changes. They’re real-time signals, not realized impact, and they can still reverse. See measurement progress for how to read a running experiment.
- Always-on optimization — adaptive policies that continuously shift allocation toward what’s working. Their value is reported against a baseline in their own block, but an optimizer that never concluded an experiment has no frozen result to accrue.
In-flight and optimization blocks, separated from realized totals
Honest limitations
The methodology is conservative, but it is still a model. Know what it assumes:- Effects are assumed to persist. A change accrues its ship-time daily impact for as long as the window runs. Novelty effects that decay, seasonality, and later product changes are not modeled.
- Independence across changes. Group intervals assume experiments err independently; interactions between overlapping changes are not modeled. Treat group intervals as indicative, not exact.
- The discount is an estimate. π is inferred from a finite history (or a cold-start default) and power is assumed or approximated — the false-positive risk is a principled correction, not a measurement.
- Projection is to 100% of the targeted population. Audience conditions and rollout caps aren’t modeled; if the winner only ever ships to a segment, the projection overstates it.
- Results freeze at decision time. A contribution restates the evidence the decision was made on; conversions still attributing when you shipped aren’t recounted later. Ratio metrics are excluded from impact aggregation.
Next steps
Significance & metric types
How the underlying lifts, intervals, and p-values are computed.
Sequential testing
Anytime-valid results — the intervals that flow into impact when you peek continuously.
Measurement progress
Reading a running experiment before it concludes.
Changes in the dashboard
The governed lifecycle that produces the concluded, evidenced decision records impact is built from.