On this page

An experiment can be carefully randomized and still be too small to answer the merchant’s question. If the store wants to detect a modest change but only reaches a few hundred eligible customers, a quiet result may reflect limited information rather than an ineffective offer.

Sample planning should begin with the smallest change that would affect the commercial decision. Choosing a target after seeing the results turns a planning assumption into a convenient explanation. The worksheet should make the tradeoff visible before customers encounter the test.

Specify the smallest useful lift

Suppose a hypothetical Shopify merchant has a 5% baseline purchase rate for the eligible population and observation window. The team wants to know whether the rebate can raise that rate to 6%. The difference is 1 percentage point, or 20% relative to the baseline. Those two descriptions refer to the same proposed change but should not be confused.

The baseline must match the planned population. A sitewide conversion rate based on sessions may not be suitable for a customer-level test limited to returning buyers. Use a comparable definition and period, and record uncertainty if the available baseline is unstable.

A minimum detectable effect is a planning input, not a promised result. It describes a change the design is intended to detect with specified probability under the assumptions. It does not guarantee detection of every smaller effect, and a larger effect may be detected with less information.

Define the primary outcome before calculating sample size. Purchase conversion, retained revenue per customer, and time to a repeat purchase have different data structures. A two-proportion calculation cannot simply be reused for a revenue metric with a highly variable distribution.

Translate assumptions into sample requirements

The University of British Columbia’s two-proportion sample-size tool provides a primary statistical reference for planning independent proportion comparisons. The calculation below is an original normal-approximation example with equal groups, a two-sided 5% significance level, and 80% power.

Worksheet input Hypothetical value Why it matters
Control purchase rate 0.05 Sets the baseline event frequency
Treatment rate worth detecting 0.06 Defines the target absolute difference
Absolute effect 0.01 Smaller differences require more information
Two-sided significance level 0.05 Specifies the planned false-positive threshold
Power at the target effect 0.80 Specifies detection probability under the assumptions
Allocation Equal groups Determines how the total sample is split
Approximate sample per group About 8,158 Independent customer assignments needed under this approximation
Rounded planning total 16,400 Operational target of 8,200 per group

Using the usual two-proportion approximation, the numerator combines the pooled-proportion term for the significance threshold with the separate-proportion term for power. For rates of 0.05 and 0.06, the resulting estimate is roughly 8,158 customers in each group. Different exact methods, continuity corrections, or design assumptions can yield a different planning value, so document the method used.

Now translate sample into time. If the store can assign 400 new eligible customers per day across both groups, reaching 16,400 assignments takes about 41 days. If each assigned customer needs 28 days of outcome follow-up, the last enrolled customers mature about 69 days after the first day of enrollment. Recruitment duration and outcome maturity are separate pieces of the schedule.

Do not count the same returning customer as a new independent assignment each day. Nor should the team use all site traffic if only part of it qualifies for the experiment. A traffic estimate that ignores those restrictions can make an infeasible test look easy.

Decide what to do when traffic is insufficient

When traffic is insufficient, change the decision design honestly. The merchant can accept that only a larger effect will be detectable, extend recruitment if the commercial conditions remain comparable, or choose a different primary outcome supported by a justified method. It should not pretend that repeated daily checks create information that the experiment does not contain.

For example, detecting a move from 5% to 7% under the same simple assumptions needs roughly 2,212 customers per group, far fewer than the 5%-to-6% design. That does not make 7% a reasonable forecast. It shows the tradeoff between the size of the effect the team wants to distinguish and the sample it can obtain.

A difficult case is a long recruitment period spanning a major seasonal change. Extending the test can increase sample size while also changing the experience being estimated. Review whether the offer, product availability, audience, and measurement remain stable enough for the intended interpretation. Time is not a free substitute for comparable conditions.

Avoid declaring “no effect” when the completed interval still contains commercially meaningful positive and negative changes. Such a result may be inconclusive for the chosen decision. The prelaunch worksheet helps the team recognize that possibility before spending the effort to run the test.

Record the final feasibility choice in one sentence: “We can recruit the planned population within the acceptable window and detect the specified effect under these assumptions,” or “This pilot will provide operational evidence but cannot reliably resolve the smaller sales effect we care about.” Both are useful conclusions.

The finished worksheet should contain the baseline source, outcome definition, target difference, statistical assumptions, eligible daily recruitment, follow-up period, and chosen action. That gives the merchant a realistic experiment timetable and a clear account of what the resulting evidence can reasonably decide.

Reserve room in the schedule for validation before exposure. A week spent repairing assignment or outcome definitions is different from a week of valid recruitment. If the team discovers after launch that only half the estimated daily visitors are new eligible customer assignments, the recruitment estimate doubles while the follow-up requirement remains. Update the feasibility worksheet promptly rather than keeping the original completion date in the campaign calendar. A planning calculation is useful only while its population assumptions still describe the test being run.

Discuss a pilot scope that matches your store volume and reporting needs. Discuss program fit.

Source references

Back to contents

General information only

This guide is general information, not financial, legal, tax or regulatory advice. Eligibility, card availability, permitted use and responsibilities depend on the applicable offer and card terms.