Sample Ratio Mismatch (SRM) Calculator

What is this?

A sample ratio mismatch (SRM) is when the number of users actually observed in each arm of your experiment doesn't match the split you configured. If you asked for 50/50 and got 5,000 vs 5,350, that gap is either random noise or a broken assignment/logging pipeline. This calculator tells you which.

The same test works for any number of arms. A three-way or four-way split gets checked all at once, and the residuals point at the arm that broke.

Why it matters

An SRM means users were added to (or dropped from) one arm in a way that isn't random. Maybe a redirect loses slow connections, a bot filter fires unevenly, or a logging bug drops events. Whatever the cause, it almost never affects both arms equally, so every metric you compute downstream is biased. The experiment results can't be trusted until the cause is found and fixed.

For the full story on what causes SRMs, real examples, and a step-by-step debugging checklist, read our guide: Sample Ratio Mismatch (SRM): what it means and how to debug it.

The test

We run a chi-squared goodness-of-fit test comparing your observed counts against the counts your configured split would produce:

\chi^2 = \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i}

Where:

$k$ is the number of arms
$O_i$ is the observed user count in arm $i$
$E_i$ is that arm's expected count under your configured split

Degrees of freedom is one less than the number of arms, so a two-way split has 1 and a four-way split has 3. The p-value is the probability of seeing a gap at least this large when the split is actually healthy. Following common SRM practice, we flag p < 0.001 as a confirmed mismatch and p < 0.01 as suspicious.

Finding the arm that broke

With three or more arms the chi-squared value tells you something is off but not where. The standardized residual answers that, one number per arm:

r_i = \frac{O_i - E_i}{\sqrt{E_i(1 - p_i)}}

Under a healthy split each residual behaves roughly like a z-score, so the arm with the largest one is where you start. Expect more than one arm to read past ±2: every arm is measured against the same observed total, so an arm that loses users makes the others look high. The biggest residual is the arm that moved on its own. With only two arms the residuals are always equal and opposite, which is why the calculator shows them from three arms up.

Calculation with Your Values

Given:

2 arms, weights 1 : 1, which is an expected split of 50% / 50%
Observed: 5,000 in A, 5,350 in B (10,350 total)

Step 1: Expected Counts

E_A = 10,350 × 0.5000 = 5,175

E_B = 10,350 × 0.5000 = 5,175

Step 2: Chi-Squared Statistic

Arm A: (5,000 − 5,175)² / 5,175 = 5.9179

Arm B: (5,350 − 5,175)² / 5,175 = 5.9179

χ² = 5.9179 + 5.9179 = 11.8357

Step 3: P-Value

Degrees of freedom = 2 arms − 1 = 1

p = P(χ² with 1 df ≥ 11.8357) = 0.00058

Step 4: Compare to the SRM Thresholds

The thresholds we use:

p < 0.001: confirmed SRM
p < 0.01: suspicious
p ≥ 0.01: healthy split

p = 0.00058, which is < 0.001, so this is a confirmed sample ratio mismatch

Step 5: How Rare This Gap Is

The p-value is the chance a healthy split shows a gap this big, so on average it takes 1 / p healthy experiments to see one:

X = 1 / p = 1 / 0.00058 ≈ 1,721

A gap this size happens by chance about 1 in 1,721 experiments.

Conclusion:

Sample ratio mismatch detected. A gap this size happens by chance about 1 in 1,721 experiments.

Sample Ratio Mismatch (SRM) Calculator

Calculator Configuration

Results

How it works

What is this?

Why it matters

The test

Finding the arm that broke

Calculation with Your Values

Found an SRM you can't explain?