Sign In

Selecting an A/B test winner

To pick a winner, you first need to determine winner criteria. A few options are: "Overall Conversion", "Revenue", or "Purchases by Product"

A Purchase is the purchase of a lifetime product or the FIRST purchase of a subscription (not including renewals). A purchase does not take into account whether a free trial was completed.

Revenue is any revenue from purchase of a lifetime product or any revenue event from a subscription (free trial conversion, renewal, etc.)

Example: If a paywall sells only products with 7-day free trials, you won't see Revenue for 7 days, until free trials convert. But you will see Purchases right away.

Learn More

Picking a Test Winner based on Overall Conversion

  1. Navigate to Insights > Paywall Conversion Rate

  2. Pick the date range the test ran

  3. Add a filter for Campaign and select ALL the test campaigns (make sure to include every placement)

  4. Segment by Paywall

  5. To rollup the data, select 'Monthly' or 'Yearly' for the Date Resolution

  6. Expand the table rows to see the individual impressions and conversions for each paywall

  7. Grab impressions and conversions for each paywall and insert into a statistical significance calculator to determine a winner.

Test not statistically significant yet?

Continue running it until you reach 97% confidence or higher

Picking a Test Winner based On Revenue

If your test includes products with a free trial, make sure to wait to capture results until after the free trials convert. Otherwise you won't see Revenue

  1. Navigate to Insights > Revenue.

  2. For the start date, choose the first day the test ran. For the End Date, choose today or the end of the test + trial length.

  3. Rollup the data by selecting 'Monthly' or 'Yearly' for the Date Resolution

  4. Add a filter for Campaign and select ALL the test campaigns (make sure to include every placement)

  5. Segment by Paywall

  6. Look at the Revenue numbers in the table to determine a winner.

Important Note

Comparing Revenue only is applicable if all variant paywalls have the same product mix. If there are different products, then the revenue differences will likely be because the products have different price points.

Picking a Test Winner based on Purchases by Product

  1. Navigate to Insights > Revenue.

  2. Pick the date range the test ran

  3. Rollup the data by selecting 'Monthly' or 'Yearly' for the Date Resolution

  4. Add a filter for Campaign and select ALL the test campaigns (make sure to include every placement)

  5. You will need to pull product purchase data separately for each paywall. To do this, filter to Paywall=Variant A.

  6. Then segment by Product SKU

  7. Download data as a .CSV file

  8. Repeat steps 5-7 for each variant in the test.

  9. Upload the CSVs into a spreadsheet

  10. Transpose and combine data into 1 table with product rows and columns for each variant.

  11. Draw conclusions based on the purchases of each product/product type

Last updated