Skip to main content
Every experiment in Traffical ends in the same question: is the difference between your variants real, or is it noise? The statistics engine answers it from the events you already collect, using methods chosen so the numbers stay trustworthy even when you check them often. This section explains, in plain terms, how that works and what each option buys you.

​
What the engine computes

For every metric on a policy, Traffical estimates each allocation’s value, the lift versus control, a confidence interval around that lift, and whether the result is statistically significant at your chosen confidence level. The same machinery serves several metric shapes:

Significance & metric types

Confidence intervals, p-values, and how binary, mean, count, and ratio metrics each get a lift and a standard error.

Sequential testing

Anytime-valid results: peek as often as you like and the false-positive rate still holds.

Variance reduction (CUPED)

Use pre-experiment data to shrink confidence intervals and reach significance sooner.

Ratio metrics

Per-session and per-order metrics where both the numerator and denominator vary.

​
Two ways to read an experiment

How you intend to look at an experiment determines which method keeps you honest.
You decide a sample size up front, wait until you reach it, and look once. This is the classic A/B test. It is the default, and it gives the tightest interval for a single planned check — but every extra peek inflates the chance of a false positive.

​
What stays true regardless

  • Your confidence level is a promise, not a suggestion. At the default 95% two-sided level, a non-significant result genuinely means the data can’t yet distinguish the variants.
  • Variance reduction never changes the answer in expectation. CUPED only removes noise that was predictable from the past — it tightens intervals without biasing the lift.
  • The engine reads aggregates, not raw rows. Everything is computed from sufficient statistics summarised in your warehouse, so analysis is cheap to re-run at any moment.
Defaults are sensible out of the box: frequentist analysis at 95% confidence, two-sided. You only need the rest of this section when you want to peek continuously, reach significance faster, or measure a ratio.