On this page

A customer-level rebate experiment can become a mixed-exposure experiment when people return on another device, share an offer, or encounter a public campaign outside the test. The original assignment may remain intact while the shopping experiences no longer differ as planned.

The aim is to understand and reduce that contamination. It is not to classify normal household behavior as fraud or to deny a customer a benefit simply because their path complicates the analysis.

Identify the likely paths between groups

Consider a hypothetical store testing a purchase reward among returning customers. One participant sees the treatment on a laptop, later shops on a phone, and opens a link sent by a family member. Another control participant encounters the same offer through a creator’s public post. The experiment’s neat two-column diagram no longer describes every exposure.

List likely paths before launch. Include repeat visits, anonymous-to-known transitions, shared devices, forwarded email, copied links, partner pages, and simultaneous campaigns. The importance of each path depends on the population and how the offer is delivered.

Separate assignment from exposure. Assignment is the experimental decision about which experience a unit should receive. Exposure is what the customer actually encounters. Keeping both records allows the team to identify a delivery gap without rewriting the original assignment history.

Do not assume a customer identifier resolves every path. Some identities cannot be joined reliably or appropriately. The protocol should describe those limits and avoid overstating the precision of its customer-level control.

Choose assignment persistence and exposure rules

Create a contamination register with a mitigation and a remaining limitation for each route. This fictional example focuses on business design rather than identity-resolution code.

Exposure path Proposed mitigation Remaining analytical limitation
Same customer returns through a recognized account Preserve the original permitted assignment Unrecognized visits may still differ
Customer changes device before signing in Use a defined rule for the transition when identity becomes available Earlier anonymous exposure may remain unknown
Treatment link is forwarded to a control customer Review the offer’s published scope and record known mixed exposure Forwarding cannot be assumed absent
Creator publishes the test offer publicly Coordinate campaign scope before the experiment Uncontrolled public exposure can weaken the planned contrast
Shared household device serves different people Avoid assuming one device equals one person Unit boundaries may remain imperfect
Another promotion overlaps the same period Document and, where feasible, control the overlapping experience The estimate may concern a combined experience

The Microsoft experimentation overview supports treating experimental validity as a designed process. The register is an original application to a merchant reward campaign; it does not imply that the platform can prevent every cross-device or shared-link exposure.

Choose the scope that the merchant can deliver credibly. A tightly controlled invitation population may answer a narrower question more reliably than a supposedly sitewide test whose offer is also advertised publicly. The narrower scope limits generalization, but that limitation can be stated honestly.

A difficult case is a customer who legitimately qualifies through a published offer despite originally being assigned to control. Apply the approved customer promise. The experiment owner should then record the deviation and assess the comparison. Preserving a spreadsheet contrast is not a reason to invent a new exclusion from a benefit.

Report contamination without silently deleting inconvenient customers

Keep the main analysis tied to the predefined assignment population where that remains appropriate. Removing every customer with mixed exposure can create a selected sample, because the likelihood of sharing, returning, or recognizing an offer may relate to purchasing behavior.

Report known contamination separately. For example, a fictional readout might say: “Of 4,000 control assignments, 120 had a recorded encounter with the treatment offer; unobserved cross-device exposure is also possible.” That is a coverage statement, not proof that the remaining 3,880 had no such encounter.

Avoid dividing the result by the observed exposure rate as an automatic correction. Estimating the effect of actual exposure rather than assignment requires additional assumptions and an appropriate method. A simple arithmetic adjustment can make a weak comparison look precise without resolving the selection problem.

Review whether contamination changes the decision the experiment can support. If the groups still had meaningfully different offer availability, the assignment effect may remain informative under the protocol’s assumptions. If both groups broadly encountered the same public offer, the test may no longer answer the intended question.

Document the customer experience actually delivered. A treatment group that saw the offer more often than control tests a difference in access or intensity, not necessarily the clean presence-versus-absence comparison originally planned. The readout should describe that distinction rather than preserve an inaccurate experiment label.

For the next test, prioritize the exposure path that caused the largest practical problem. That might mean coordinating creator activity, clarifying invitation scope, or changing the assignment unit. It does not necessarily require collecting more personal information or treating shared devices as suspicious.

The finished contamination register should make the remaining uncertainty visible before results are interpreted. A useful experiment can have limitations; the failure is claiming that every assigned customer received a perfectly isolated experience when the merchant’s own campaign routes show otherwise.

Distinguish a customer seeing the offer from actually receiving a reward. A control participant may encounter a shared advertisement without buying anything. Another may complete a qualifying purchase under a valid published route. Those are different deviations with different customer consequences. Record the observed exposure and the customer’s actual approved benefit separately. Collapsing both into a single “contaminated” flag hides the mechanism and can encourage inappropriate case handling. The experiment report needs the exposure limitation; the service record needs the correct customer promise.

Discuss how your Shopify customer journey could support consistent treatment assignment. Explore Shopify integration.

Source references

Back to contents

General information only

This guide is general information, not financial, legal, tax or regulatory advice. Eligibility, card availability, permitted use and responsibilities depend on the applicable offer and card terms.