Sample ratio mismatch (SRM): what it is and how to fix a broken A/B test split

An experiment result that everyone believes and nobody can trace back to the product is the most expensive number a growth team can produce. One way it happens is when the two groups being compared quietly stopped being the same size, and nobody counted. I'll walk one online store's checkout test from setup to result, so you can see the break happen and see the ten seconds of arithmetic that catches it.

Quick answer: Sample ratio mismatch (SRM) is when the number of visitors who actually ended up in each group of an A/B test is further from the split you asked for than luck can explain, which is a warning that something in the machinery that sorts visitors into groups, shows them a version, records what they did, or counts them afterwards has changed who ended up on each side. Until you know what, you can't treat the comparison between the two groups as fair, or say with confidence that your change caused the result. Every term in those two sentences, including A/B test, split, lift, conversion, randomization, and the statistics check that detects the problem, gets broken down step by step below.

How an A/B test is supposed to work: one store, two checkout pages

An A/B test earns its trust from one thing: a coin flip, not a person, decides which version each visitor sees. Picture an online store with an ordinary complaint. Plenty of visitors fill a cart and then give up on the checkout page, so the team builds a simpler one, fewer fields and a single screen instead of three.

Now there are two versions of the same page. The existing checkout page is the control, the version they already have and want to beat. The simpler page is the variant, the new thing being tried. To judge which one wins, the team counts the share of visitors who finish a purchase, and each finished purchase is a conversion.

The team tells their testing tool to send half of arriving visitors to the control and half to the variant. That instruction is the 50/50 split, and running two versions side by side and comparing what each half does is an A/B test. For every visitor who arrives, the tool effectively flips a coin, and that flip, not anything about the person, decides which page they get. The coin flip is called randomization.

Randomization is the whole reason the comparison means anything. Because the coin decides and nothing about the visitor does, across enough visitors the two groups come out comparable on everything you thought to measure and everything you didn't: roughly the same mix of new and returning customers, of phones and laptops, of patient and impatient people. The one thing that differs by design is which checkout page they saw. That is what lets you put a difference in purchases down to the page rather than to who ended up where, once the difference is big enough to stand out from ordinary wobble. One note on counting, since it matters later: I'll say visitors throughout, but count whatever your test actually sorts, which is usually unique people rather than individual visits.

Two weeks later, the new page looks 4% better

The test runs for two weeks, 10,000 visitors go through it, and the dashboard says the simpler checkout page converts about 4% better than the old one. That difference is the lift, the number the whole exercise exists to produce.

The team is thrilled. The simpler pattern gets scheduled for the signup flow too, the 4% goes in a deck, and the next quarter of roadmap gets written on the strength of it.

Is a 5,210 / 4,790 split unlucky or broken?

Then an analyst asks a boring question: how many visitors actually ended up in each group? It was supposed to be 5,000 and 5,000. It was 5,210 and 4,790.

A gap of 210 sounds like nothing. It sounds like the sort of wobble you'd expect from flipping a coin 10,000 times. Coin flips do wobble, but within a predictable range. At 10,000 flips, the two sides landing 100 or so apart is ordinary. Landing 420 apart, which is the real distance between 5,210 and 4,790, is not.

The standard way to settle this is a statistics check called chi-squared. It answers exactly one question: could a gap this size happen by pure luck? You hand it the two counts you expected and the two counts you actually got, and it hands back a probability: how unlikely a gap this large or larger would be if the split you asked for were working correctly. In the calculation below, each half of the sum compares one group's expected count against what it really got, and the larger the total, the worse luck looks as an explanation.

expected:  5,000 vs 5,000     observed:  5,210 vs 4,790

chi-squared = (5210-5000)^2/5000 + (4790-5000)^2/5000 = 17.6
p = 0.00003  (about 1 in 37,000 by chance)

This is a sample ratio mismatch: if the intended even split were working correctly, a gap this large or larger would turn up in about one experiment in 37,000. Something between the coin flip and this table is inconsistent with the split you asked for, so the comparison between the two groups can't be taken at face value until you know what.

Notice what that check never looks at: how many people bought anything. It compares group sizes and nothing else. A test whose sales numbers look completely reasonable can still fail it, which is why group sizes have to be checked on their own. If you want to see where the line sits before you go arguing about a live test, our free SRM calculator runs this check on any two numbers.

What the broken split means for the store's 4%

That imbalance did not come from the coin flip alone. Something changed who got counted on one side, and whatever it was had a reason, which means the two groups being compared may no longer be alike. Notice the counts don't even tell you how many people are unaccounted for: 5,210 against 4,790 is equally consistent with visitors dropping out of one group, with extra records piling up in the other, or with both happening at once.

Say the simpler checkout page is heavier and takes an extra second and a half to appear. Some visitors on a weak connection give up and close the tab before it finishes loading, and because a visitor only gets recorded once the page actually appears, those people never get counted at all. The variant group has quietly had its least patient visitors stripped out. The control group still has all of its.

That is selection bias: the people who dropped out weren't a typical cross-section, so what's left on each side is no longer comparable. The simpler page now looks better partly because the twitchiest visitors were removed from its group, and nothing in the data tells the team how much of the 4% came from the page and how much came from who disappeared. The result isn't proven fake. It's unattributable, which for a decision is just as bad, because nobody can point at what they shipped and say it did this.

I've seen teams celebrate a result for weeks before anyone counted the two groups. By the time it surfaced, the feature was fully built out and two quarters of roadmap were written against a number nobody could trace back to the product.

This happens often enough to plan for. Microsoft's experimentation team found this problem in roughly 6% of their experiments. At a similar rate, a team running 40 tests a year would hit two or three results they can't attribute to anything, with no way of knowing which ones without checking.

How do you know you have an SRM?

You run that chi-squared check on the two group counts, and in most cases something runs it for you. Many major experimentation platforms now monitor for this automatically, though the method, the threshold, the timing and which experiments are covered all vary between them, so the practical skill is knowing what your own platform's flag means rather than doing the arithmetic yourself.

If you have two group counts in front of you right now, check them here.

Two things about the check are worth knowing before you act on it. First, the bar is set deliberately high. That luck probability, written as a p-value, usually has to fall somewhere around one in a thousand before a platform sounds the alarm, rather than the one in twenty you may have seen used elsewhere. At one in twenty, one perfectly healthy test in every twenty would trip it, and within a month your team would be ignoring the alarm entirely. Eppo runs this check at one in a thousand and gives a plain reason for staying conservative: the check gets run over and over as data arrives, so a strict bar keeps false alarms down while still catching very nearly every real break.

Second, the check gets more sensitive as the test gets bigger. Using the stricter one-in-two-thousand bar that Microsoft's experimentation team runs, a gap of about 350 trips it at the store's 10,000 visitors. At 200,000 visitors, a gap of about 1,560 trips it, and that works out to 50.4% against 49.6%. A split that reads as fine on a dashboard rounded to one decimal place can be badly broken underneath.

How bad is it? Severity is not the size of the imbalance

The size of the gap doesn't tell you how big the bias in your result is. The check answers one question, whether the imbalance is bigger than the split you expected can explain, and it makes no attempt to measure how much your result got twisted.

A half-percent gap caused by tracking that quietly fails on one version can distort a sales number far more than a three-percent gap caused by someone bumping the traffic setting on a Tuesday. What determines the damage is the cause, specifically which visitors disappeared and how they differ from the ones who stayed. At the moment the alarm fires, the store's team doesn't know the cause, so they know neither the size of the distortion nor which direction it pushes their 4%.

That leaves no version of this small enough to wave through on size alone. Once the check fires, stop interpreting the result until you know what caused the imbalance. Sometimes the cause turns out to be benign, or confined to one segment, and the experiment survives. You cannot know which until you look.

What causes sample ratio mismatch

Almost every broken split I've chased comes from one of five places in the chain between a visitor arriving and a number landing on a dashboard. They're listed in the order I'd check them, which is roughly cheapest to most expensive. The first one you can often answer by reading a settings screen. The last one needs someone to go looking through what else was running that week.

Eligibility: who was allowed into the test

Eligibility is the rule deciding which visitors belong in the test at all. Our store might keep its own staff out of the checkout test, or keep out wholesale buyers who check out through a different flow entirely.

The rule is never the problem. Running it late is. If the coin flip happens first and the exclusion happens second, both groups fill up with people who were never meant to be there, and then those people get removed unevenly. Say three in ten visitors are wholesale buyers. Ten thousand visitors get split into 5,000 and 5,000. The wholesale buyers sorted into the new checkout get sent to the old one instead, because the new page was never meant for them, so they never register as new-checkout visitors. The wholesale buyers sorted into the old checkout were always going to see it, so they stay counted. You report 5,000 against 3,500, and the coin flip was perfect the whole time.

This is the most common cause I run into, and it's the cheapest to confirm: ask whether any rule that removes people runs after the split. The fix is to decide who qualifies first and flip the coin second. The refinement that saves you months later is to give the people who don't qualify their own label instead of dropping them, so you can show they were excluded on purpose rather than lost.

Assignment: the coin flip that sorts them

Assignment is the moment a visitor gets sorted into a group and recorded as belonging to it. Everything downstream inherits whatever happens here, which is why it's second on the list and not tenth.

Four things break it in practice. The identity a visitor gets sorted by can change underneath them, most often when someone browses while logged out and then signs in, so the same shopper is sorted twice and lands on both sides. Someone can change how traffic is divided partway through the run, so Monday's visitors were split differently from Thursday's. A manual override left switched on in one environment quietly forces a group of people into one version. And when two tests run at the same time, their coin flips should be independent of each other. Reusing the same sorting recipe for both ties them together, so landing in one test's new version predicts landing in the other's. That alone doesn't bend either split, but it tangles the two results together, and it sets up the failure in the next section, where the combination itself is what does the damage.

Compare the split you configured against the split your testing tool reports, then find the day they diverged. A gap that starts on day three points at a release or a settings change that day. A gap present in the first hour points at how visitors are being sorted.

Exposure: whether they saw it, and whether that got written down

Exposure is the record that a visitor actually saw the version they were assigned. Assignment says what they were supposed to get. Exposure says what they really got, and when.

This is where a fair coin flip still produces a broken split. The new checkout might fail to load, and because a visitor only counts once the page appears, they are never recorded at all. A stored copy of the old page, kept close to the visitor so it loads faster, might get handed to someone who was assigned the new one. Or the code that writes the record lives inside the new page and not the old one, so one group's records arrive and the other's quietly don't.

There's a version of this that catches people out. If the new page is faster, more of its records survive the trip, and it can show more visitors for no reason except that it improved. A split can break because something got better.

The check is to compare how many visitors were assigned against how many were recorded as seeing it, one group at a time. When those two numbers disagree in one group and match in the other, you're here.

Data pipeline and analysis: what happens to the records after they land

This covers everything between the records arriving and the number you read, and that's more than the queries an analyst writes. Before anyone writes one, an automatic cleanup runs: it strips out visits from bots and crawlers rather than people, stitches records together, and waits on data that arrives late.

Any of those can hit one group harder than the other. The example I find most instructive is a test where the most engaged visitors on the new version clicked so much that the bot filter decided they were robots and removed them. The result flipped once somebody noticed. Nobody had touched the experiment, and the number describing it moved anyway.

Then there are the analyst's own choices, like a filter applied unevenly or a lookup that quietly drops anyone who never bought. Count the visitors at each hop, from your testing tool through to the table your dashboard reads, and find the hop where the two groups stop agreeing.

Runtime interference: someone changed the test while it was running

The last place to look is outside the experiment itself. Somebody pauses one version for an afternoon and resumes it. Somebody ramps one version up before the other. A marketing campaign links straight to one version's address, pouring outside traffic into one side of a test that was supposed to be split by coin flip. None of these are bugs, which is exactly why they survive review, and all of them bend the ratio.

Running another test at the same time is worth checking here too, though it costs you differently than people expect. Two independently sorted tests don't bend each other's splits, and on most platforms a visitor can be in several tests at once, so you don't lose sample either. You lose sample when tests are deliberately kept apart from each other, or when a slice of visitors is held back from new changes entirely so the company can measure long-run effects. What bends a split is the combination misbehaving: if being in both new versions at once slows or breaks the page, records go missing from that one combination.

There's still a second question worth asking when tests overlap, and it isn't about counts at all. Two tests can leave your split perfectly balanced and still change each other's results, so the lift you measure for one depends on which version of the other a visitor happened to see. Our free interaction effects calculator checks for that.

What the team does when the check fires

Freeze the decision before touching anything else. The store's team doesn't ship the simpler checkout, doesn't put the 4% in the deck, and doesn't extend the test hoping the ratio heals. If the cause is systematic, a longer run just compounds it. And if an early imbalance later washes out of the running totals, that hides what happened rather than explaining it.

Then they find the cause. The two headline counts tell almost nothing on their own. The job is to break them apart until the imbalance stops being everywhere and starts being somewhere specific.

Each of those answers hands the problem to a specific person. A day-three break is a question for whoever released on day three. A Safari-only break belongs to a front-end engineer. A flag-versus-warehouse gap belongs to data engineering. That's the point of slicing the numbers: the team stops debating whether the 4% is real and starts deciding who owns the fix. Once the cause is fixed, what happens next depends on what was actually lost. If visitors were never recorded at all, that data can't be reconstructed and the test gets rerun. If the records exist and the damage happened later, in a filter or a join or a batch that arrived late, the pipeline can often be corrected and the same data recomputed instead.

One explanation is routinely benign: rounding. A platform that can only allocate whole percentage points can't give you a true 33.3/33.3/33.3 across three versions, so the counts drift by a fraction of a percent with nothing broken behind them. Compare the split you configured against the split you got, and if the gap matches the rounding, the result stands. Other explainable cases exist, but none of them are safe to assume before you have found the cause.

One more looks innocent and isn't. Suppose visitors get sorted into groups earlier in the funnel, and the team then counts only the ones who reached the checkout page. If the version being tested changes whether people get that far, then the group being counted was filtered after the coin flip by something the test itself caused. That is a real bias rather than a false alarm, and the fix is to count visitors at the moment they are sorted into groups instead of at the moment they arrive at the page. Everything else stays a stop and investigate.

Manual SRM checks vs what your platform already does

Most teams have some automatic checking in place and assume it covers them. It covers less than people think, and the gap is worth understanding before you rely on a green checkmark in a vendor dashboard.

That last row is where teams get burned. A green checkmark only covers the population and the data that the check actually ran on. If your analysts rebuild the checkout numbers somewhere else with a different lookup, a different bot filter, or a different date cutoff, that checkmark says nothing about the figures in your board deck. Run the check against the table you actually report from.

Common pitfalls

Checking only at the end is the most expensive habit. A broken split caught on day two costs two days of traffic. The same break caught at the end may cost the whole test plus a full re-run, and you still have to explain to whoever was waiting why there's no answer.

The second pitfall is trying to fix the ratio by deleting visitors. Dropping 420 records from the larger group gives you a neat 4,790 against 4,790, and it repairs nothing. Equalizing the counts afterwards doesn't undo whatever process created the imbalance, and it doesn't restore the coin flip that the comparison depends on. You've matched two numbers without recovering a single one of the people the comparison actually needs.

The third is the hardest to enforce. Twyman's law says any figure that looks unusually interesting is usually wrong, and a suspiciously clean 4% on a broken split is the most expensive version of that, because nobody in the room has any reason to question it.

FAQ

What is sample ratio mismatch in an A/B test?

Sample ratio mismatch is when the number of visitors who actually landed in each version of your test differs from the split you configured by more than chance allows. It points to a possible problem in how visitors were sorted, which version they were shown, how their behavior was recorded, or how the numbers were assembled afterwards. Until you know the cause, you can't attribute the difference between the two groups to the change you shipped.

How do I check for SRM?

Compare the actual visitor count in each group against the split you asked for, and run a chi-squared check on those two numbers to see whether the gap is plausible by luck. Most experimentation platforms now run this for you automatically, though how often they check and which experiments they cover varies, and our free SRM calculator will do it in seconds if yours doesn't.

Can I still use the result if the imbalance is small?

Not until you know why it happened. Size alone is no reason to relax, because a 0.5% imbalance from tracking that fails on one version can distort a conversion rate more than a 3% imbalance from a traffic setting someone nudged. If you trace it to a cause that leaves both the coin flip and the analysis intact, the result may still stand.

Does SRM happen in server-side tests too?

Yes. The usual causes on the server are different machines sorting the same person into different groups, the recording step sitting inside a code path only one version ever reaches, and targeting rules that read information the new version has already changed.

How do we prevent SRM before launch?

Run a dry run in which both groups get the identical page, and confirm the counts that come back are consistent with the split you asked for, through your real assignment path. Make sure no rule that removes people runs after the coin flip. Check that the counts your feature flag tool reports match the counts in your warehouse before you start, freeze the traffic allocation for the full run, and watch the split from day one instead of only at the end. Use your platform's own ongoing monitoring for that rather than re-running a one-shot check every day, because repeating the same test over and over raises the odds of a false alarm.

If your team has never run this check, that's the first thing I'd look at, because it tells you whether the results you've already acted on were ever attributable to the product at all. Happy to walk through what you're seeing on a 15-minute call.