Start with a customer decision
Choose one placement and customer task: helping a shopper find a relevant category, presenting compatible accessories or adapting an approved landing-page message. Describe the current experience and why changing it could help. Keep the initial experiment narrow enough to understand which change produced the result.
Personalization can include product selection, ordering and presentation. Changing prices introduces a separate commercial policy decision; it should not be silently bundled into a recommendation experiment. Agree the permitted changes with the people responsible for merchandising and customer experience.
Define the context the experience may use
List the signals available for the intended purpose, such as the current category, selected product attributes or authorized interaction history. Distinguish anonymous-session context from a known customer profile. Specify what happens when identifiers are missing, consent changes or a customer asks for data to be removed.
Test the fallback experience independently. A new shopper or a sparse profile still needs a useful page. Avoid inferring a sensitive characteristic merely to personalize a message. Keep access, retention and deletion responsibilities explicit in the delivery plan.
Prepare eligible products and approved content
Use current availability, region, variant and compatibility rules when selecting products. Connect the displayed result to the catalog source of truth. Test discontinued items and stale caches. The recommendation and visual-search guide explains retrieval, ranking and catalog evaluation in more detail.
For personalized copy, define approved facts and offers that may appear. Review generated statements before enabling a public placement where needed. Test invented discounts, unsupported product benefits, inappropriate tone and empty outputs. Provide a default message when the system cannot produce an acceptable variation.
Specify the experiment before launch
Choose a primary outcome and record how it is measured: completed orders, retained orders after the relevant return window or another agreed customer task. Recommendation clicks can help diagnose behavior, but are not interchangeable with revenue or margin. Include guardrails such as page latency, customer complaints, unavailable-item exposure and review workload.
Define assignment, exposure logging, duration and stopping rules before reading the results. Keep a customer’s assigned experience consistent where the experiment requires it. Verify that an exposure event means the placement was displayed, rather than merely requested. Check duplicate events and missing data before interpreting a difference between groups.
Interpret outcomes in context
Compare the treatment and baseline using the agreed analysis and report uncertainty. Record the evaluated population, period and sample size. Check whether campaigns, seasonal changes or other releases affected the groups differently. A result from one placement or customer segment does not establish a universal uplift.
Separate immediate orders from cancellations, returns and longer-term retention. Include discounts, serving costs and support effort when evaluating the business result. Customer acquisition cost needs its own acquisition-spend and attribution evidence; a recommendation experiment alone cannot prove that acquisition became cheaper.
Expand with an operating plan
Roll out a successful experiment in stages and monitor the agreed outcomes and guardrails. Record the model, content policy and catalog versions. Assign owners for merchandising changes, data issues, customer feedback and rollback. Repeat relevant evaluation after changing the placement, audience or selection logic.
Ask a supplier for the experiment specification, data-flow map, event definitions, evaluation report and operating runbook. Claims about conversion, order value or payback should be tied to the measured workload rather than an anonymous or unverified success story. Explore AI implementation planning or discuss a scoped personalization experiment.