On this page

A test configured as 50/50 does not need to produce exactly equal counts. Random allocation naturally varies. But a material mismatch between the expected and observed groups can signal that the experiment lost or selected customers in a way that also distorts the sales result.

The correct response is to investigate the population before interpreting the winner. A promising conversion chart does not make a broken assignment or filtering process trustworthy.

Compare expected and observed allocation

Start with the planned allocation and the correct counting unit. If the experiment assigns customers, compare assigned customer counts. Order counts can differ because the treatment changes purchasing behavior; that difference is an outcome, not automatically a sample-ratio problem.

In a hypothetical test, the assignment log contains 5,000 treatment customers and 5,000 control customers. The analysis table contains 4,500 treatment customers and 5,000 controls. The discrepancy appears after assignment, so the investigation should focus on what happened between the log and the analysis population.

For an equal split of the 9,500 analyzed customers, expected counts would be 4,750 in each group. A simple chi-square allocation statistic is approximately 26.32: the two squared deviations of 250 are each divided by 4,750 and then added. The calculation is a diagnostic signal under the relevant assumptions, not an explanation of the cause.

Microsoft’s sample-ratio mismatch guidance describes checking imbalance and tracing failures across experiment stages. Its central practical lesson here is to diagnose the mismatch before trusting the effect estimate. The merchant should choose its own documented detection procedure rather than copy an arbitrary percentage tolerance.

Trace where the population became unequal

Trace the population through a sequence of checkpoints. Keep counts at each stage, using consistent identifiers and the intended allocation for that stage.

Checkpoint Hypothetical observation Question it changes
Eligible population before assignment Eligibility rule applied before exposure Was selection independent of treatment?
Assignment log 5,000 treatment; 5,000 control Did initial allocation match the design?
Exposure log 4,800 treatment; 4,900 control observed Did the experiences affect observation or delivery?
Joined outcome table 4,500 treatment; 5,000 control Did a join remove customers without outcome records?
Final exclusions Same 4,500 and 5,000 Were later filters applied consistently?
Reported conversion table Only customers with a recognized reward event retained Does the analysis condition on treatment engagement?

The last row reveals a plausible error in this fictional investigation: the treatment group was filtered to a reward-related event that has no equivalent meaning in control. The experiment no longer compares all assigned customers. A correction must restore the intended population from reliable records, not simply force the counts to look equal.

Inspect missing records by relevant pre-exposure characteristics and time. A mismatch concentrated after a site change may point to a new logging or join behavior. A mismatch concentrated in one browser can suggest an observation issue. These slices help locate the problem; they are not an invitation to discard the affected slice until the result improves.

A difficult case is an intentionally unequal rollout. If the merchant assigned 10% of early customers to treatment and later increased the share, the expected allocation must reflect that actual design. Comparing the pooled data with 50/50 can create a false alarm. Reconstruct the intended counts by assignment period and evaluate the design actually used.

Quarantine results until the mismatch is explained

Quarantine the affected sales conclusion while the investigation is open. This means labeling the result unsuitable for the planned causal decision, not hiding the operational incident. The business can still respond to customer problems or stop an unsafe experience without pretending that the experiment has produced a valid effect estimate.

Write a short investigation record with the first failing checkpoint, affected period, suspected mechanism, evidence, correction, and remaining uncertainty. For the fictional join error, the correction might rebuild both groups from the immutable assignment list and left-join outcomes so that a missing purchase record does not remove a customer.

Then rerun the population checks using the corrected data. Matching counts alone are insufficient. The correction should address the identified mechanism and preserve the intended outcome definitions. Artificially dropping control customers until the groups have equal size merely conceals the mismatch and may add new selection bias.

If the missing information cannot be recovered, state that limitation. A fresh experiment may be more defensible than a retrospective repair based on assumptions that cannot be checked. Document which operational lessons remain useful and which commercial claims cannot be made from the affected test.

Keep the detection check available before future readouts. It should be visible at the point where analysts and decision-makers first see the experiment’s result. A warning buried in a separate data-quality report is easy to overlook when a dashboard announces a winner.

The completed investigation should explain why the population became unequal and why the proposed correction restores, or fails to restore, the planned comparison. A sample-ratio check is valuable because it protects the meaning of the result, not because every experiment must display identical group counts.

One useful diagnostic is to compare a deliberately simple outcome-free table with the final analytical table. Start from the assignment list and retain every permitted customer identifier without joining purchase or reward records. If that population matches the intended allocation but the final table does not, the later data preparation deserves attention. This comparison does not prove assignment was flawless, but it narrows the search. It also prevents a team from spending days changing the randomization process when the actual defect is a filter that removes customers with no observed purchase.

Review the campaign data available through Rebate Card X before drawing conclusions from an imbalanced test. Explore capabilities.

Source references

Back to contents

General information only

This guide is general information, not financial, legal, tax or regulatory advice. Eligibility, card availability, permitted use and responsibilities depend on the applicable offer and card terms.