Skip to main content
Every measurement in Traffical ends in the same question: is the difference between your allocations real, or is it noise? The statistics engine answers it from the events you already collect, using methods chosen so the numbers stay trustworthy even when you check them often. This section explains, in plain terms, how that works and what each option buys you.

​
What the engine computes

For every metric on a policy, Traffical estimates each allocation’s value, the lift versus control, a confidence interval around that lift, and whether the result is statistically significant at your chosen confidence level. The rest of this section covers that machinery and what’s built on top of it:

Significance & metric types

Confidence intervals, p-values, and how binary, mean, count, and ratio metrics each get a lift and a standard error.

Sequential testing

Anytime-valid results: peek as often as you like and the false-positive rate still holds.

Variance reduction (CUPED)

Use data from before first assignment to shrink confidence intervals and reach significance sooner.

Ratio metrics

Per-session and per-order metrics where both the numerator and denominator vary.

Measurement progress

How much longer does this need? The target effect, what’s detectable right now, and the estimated time remaining.

Impact methodology

Program-level impact: shipped wins projected to rollout, discounted for false-positive risk, carried with intervals.

​
Two ways to read a result

How you intend to look at the results determines which method keeps you honest.
You decide a sample size — a target count of enrolled units — up front, wait until you reach it, and look once. This is the classic A/B test. It is the default, and it gives the tightest interval for a single planned check — but every extra peek inflates the chance of a false positive.

​
What stays true regardless

  • Your confidence level is a promise, not a suggestion. At the default 95% two-sided level, a non-significant result genuinely means the data can’t yet distinguish the allocations.
  • Variance reduction never changes the answer in expectation. CUPED only removes noise that was predictable from the past — it tightens intervals without biasing the lift.
  • The engine reads aggregates, not raw rows. Everything is computed from sufficient statistics summarised in your warehouse, so analysis is cheap to re-run at any moment.
Defaults are sensible out of the box: frequentist analysis at 95% confidence, two-sided. Reach for the rest of this section when you want to peek continuously, reach significance faster, measure a ratio, estimate how much longer a measurement needs, or roll shipped wins up into a program-level impact number.