Evaluating a Shopify quantity-tier test with conversion, AOV, and revenue per visitor

What was happening

A Shopify product page already offered quantity discounts at 3, 6, 10, and 20 units. A new tier presentation changed how customers selected quantity and understood the bundle offer. Both experiences received similar traffic: 14,672 visitors reached the live experience and 14,951 reached the variation. The test produced 106 orders and $12,516.62 for the live experience compared with 100 orders and $10,377.81 for the variation.

Why the obvious solution was not enough

A merchandising change can increase interaction while reducing order value, or improve conversion while attracting smaller baskets. The test therefore could not be judged from tier clicks, add-to-cart activity, or conversion rate alone. Traffic also needed to stay separated consistently so the result was not distorted by customers switching experiences during the same journey.

How the solution works

1. Define the customer-facing difference before launch and keep price rules, product eligibility, inventory behavior, and checkout logic consistent between cohorts.
2. Split traffic between the current experience and the variation, then preserve the assigned experience throughout the product and cart journey.
3. Track visitors, orders, revenue, conversion rate, average order value, and revenue per visitor for each experience.
4. Use revenue per visitor as the combined business signal because it reflects both the likelihood of purchase and the value of the resulting order.
5. Review conversion and average order value separately to understand why revenue per visitor moved instead of treating the combined metric as a black box.
6. Check the quantity tiers for inventory limits, sale pricing, mobile readability, and cart consistency so implementation errors do not masquerade as customer preference.
7. Stop the test with one documented decision. Keep the stronger experience, remove the losing variation from the active test, and avoid layering the next pricing experiment on top of unresolved code.

const live = {
  visitors: 14672,
  orders: 106,
  revenue: 12516.62,
};

const variant = {
  visitors: 14951,
  orders: 100,
  revenue: 10377.81,
};

function summarize(cohort) {
  return {
    conversionRate: cohort.orders / cohort.visitors,
    averageOrderValue: cohort.revenue / cohort.orders,
    revenuePerVisitor: cohort.revenue / cohort.visitors,
  };
}

compare(summarize(live), summarize(variant));

What should be verified before shipping

• Confirm cohort assignment remains stable from landing through checkout and does not reset when a customer returns to a product page.
• Verify the same products, prices, tier thresholds, exclusions, and stock rules are available to both experiences.
• Compare order counts with Shopify orders and revenue rather than relying only on browser-side events.
• Inspect conversion rate, average order value, and revenue per visitor together before making the decision.
• Test desktop and mobile separately for broken tier states, unreadable totals, or quantity controls that could bias one cohort.
• Record the final decision and stop the losing experience so later analysis does not mix traffic from different test periods.

What changed

The live experience recorded a 0.72% conversion rate, $118.08 average order value, and $0.85 revenue per visitor. The variation recorded 0.67%, $103.78, and $0.69 respectively. The live experience therefore produced about 23% more revenue per visitor and about 14% higher average order value during the observed test period, so it was retained and the variation was stopped. These figures describe the observed test result and are not presented as a guarantee that the same lift would repeat in another store or period.

Reusable lesson

A conversion test should answer a business question, not reward the newest interface. Evaluate the complete purchase outcome, keep the implementation variables controlled, and let revenue per visitor show whether conversion and basket value improved together.