On this page
A store’s sales can rise after a rebate launch even when the offer causes none of the increase. A seasonal demand change, product restock, or advertising expansion may have happened at the same time. Comparing “before” with “after” gives the whole change to the launch unless the analysis addresses those other influences.
A comparison group can help, but it does not automatically create an experiment. Difference-in-differences estimates a change relative to another change. Its credibility depends on whether that comparison represents what would plausibly have happened without the reward offer.
List what changed alongside the rebate
Start with a confounder log. Record changes in product availability, prices, acquisition spend, audience composition, store experience, and external demand that could affect the outcome around launch. The list should reflect actual evidence, not every imaginable explanation.
In a fictional example, a homeware merchant introduces a rebate in one comparable customer market while another market continues without the new offer. Both markets enter a seasonal buying period. The treatment market also receives a new product delivery shortly after launch, which could create additional demand independently of the reward.
Use an outcome with a consistent definition in both groups and periods. Suppose the measure is retained revenue per eligible customer over a defined period. A comparison based on gross orders in one market and retained revenue in the other would fail before any statistical method is applied.
| Potential concurrent change | Fictional evidence | Implication for interpretation |
|---|---|---|
| Seasonal demand | Both markets enter the same buying period | A comparison may help remove a shared change |
| New stock delivery | Confirmed only in the treatment market | Could explain part of the apparent reward effect |
| Advertising budget | Recorded as unchanged in both markets | One known alternative explanation is less likely |
| Customer mix | New-customer share rose in treatment | Requires review of whether populations remain comparable |
| Store measurement change | No documented change | Still verify the data rather than relying only on release notes |
Test whether the comparison trend is credible
The World Bank’s discussion of parallel trends explains a central challenge of this method: similar-looking earlier trends do not prove that the groups would have continued to change similarly without treatment. Pre-period evidence is useful, but the counterfactual assumption remains substantive.
Examine several earlier periods where available. Ask whether the groups moved in comparable ways, whether they respond similarly to seasonal demand, and whether the launch period introduced a group-specific shock. A single pre-period cannot reveal a trend by itself.
Now consider this hypothetical calculation:
| Group | Before revenue per eligible customer | After revenue per eligible customer | Change |
|---|---|---|---|
| Reward-offer market | $40 | $52 | +$12 |
| Comparison market | $38 | $46 | +$8 |
| Difference in changes | +$4 |
The raw treatment increase is $12. Subtracting the comparison increase of $8 gives a difference-in-differences point estimate of $4 per eligible customer. The method attributes the remaining difference to the intervention only under the relevant assumptions. It does not magically determine which part came from the new stock delivery.
A difficult case arises when the reward itself changes customer composition. Conditioning the analysis on a post-launch mix can remove part of the effect or introduce selection. Decide the population and analytical approach in a way that matches the causal question, rather than adjusting every observed difference until the groups look alike.
Present an estimate with explicit causal limitations
Write the result with its evidence grade. A defensible description of the fictional calculation is: “Revenue per eligible customer increased by $4 more in the reward market than in the comparison market; the estimate remains exposed to treatment-specific stock and audience changes.” It is not yet “the rebate generated $4 per customer.”
Add uncertainty using a method appropriate to the design and unit of variation. If treatment was introduced at market level, thousands of customer records do not necessarily provide thousands of independent treatment assignments. The analysis needs to respect the structure that generated the comparison.
Review alternative plausible specifications as sensitivity analysis, not as a search for a favorable answer. For example, examine whether the estimate changes materially when periods affected by a documented stock interruption are treated consistently under a justified rule. Report why the alternative is relevant and what it changes.
If the comparison cannot be defended, retain the descriptive finding and stop short of a causal claim. “Sales rose after launch” can still be useful operational information. The limitation is that it does not establish how much the rebate caused.
Use the confounder log to design a better next measurement. A future holdout may be possible before the next offer is shown. Alternatively, the merchant may need a more comparable rollout or better pre-period data. The next step should address the specific ambiguity the observational analysis exposed.
The completed output is a worked change table, a parallel-trend review, a concurrent-change log, and a clearly bounded claim. That package lets a decision-maker see both the apparent commercial movement and the reasons it cannot automatically be credited to the reward campaign.
Run a placebo timing check when the data and design make it appropriate: apply the same descriptive change calculation to an earlier period in which the offer had not yet launched. A large apparent effect there is a reason to investigate the comparison’s stability. A small placebo difference is reassuring only within its limits; it cannot rule out a new treatment-specific shock at the real launch. In the fictional example, the stock delivery remains a live concern even if earlier periods look similar. Record that distinction so that a passed historical check does not become a blanket certificate of causality. The business review should ask which plausible alternative explanation remains large enough to change the decision.
Explore reporting capabilities that can help structure a cautious rebate launch readout. Explore capabilities.
Source references
This guide is general information, not financial, legal, tax or regulatory advice. Eligibility, card availability, permitted use and responsibilities depend on the applicable offer and card terms.
