Selecting an A/B test winner
Last updated
Last updated
To pick a winner, you first need to determine winner criteria. A few options are: "Overall Conversion", "Revenue", or "Purchases by Product"
Picking a Test Winner based on Overall Conversion
Navigate to
Pick the date range the test ran
Add a filter for Campaign and select ALL the test campaigns (make sure to include every placement)
Segment by Paywall
To rollup the data, select 'Monthly' or 'Yearly' for the Date Resolution
Expand the table rows to see the individual impressions and conversions for each paywall
Grab impressions and conversions for each paywall and insert into a statistical significance calculator to determine a winner.
Test not statistically significant yet?
Continue running it until you reach 97% confidence or higher
Picking a Test Winner based On Revenue
For the start date, choose the first day the test ran. For the End Date, choose today or the end of the test + trial length.
Rollup the data by selecting 'Monthly' or 'Yearly' for the Date Resolution
Add a filter for Campaign and select ALL the test campaigns (make sure to include every placement)
Segment by Paywall
Look at the Revenue numbers in the table to determine a winner.
Important Note
Comparing Revenue only is applicable if all variant paywalls have the same product mix. If there are different products, then the revenue differences will likely be because the products have different price points.
Picking a Test Winner based on Purchases by Product
Pick the date range the test ran
Rollup the data by selecting 'Monthly' or 'Yearly' for the Date Resolution
Add a filter for Campaign and select ALL the test campaigns (make sure to include every placement)
You will need to pull product purchase data separately for each paywall. To do this, filter to Paywall=Variant A.
Then segment by Product SKU
Download data as a .CSV file
Repeat steps 5-7 for each variant in the test.
Upload the CSVs into a spreadsheet
Transpose and combine data into 1 table with product rows and columns for each variant.
Draw conclusions based on the purchases of each product/product type
Navigate to
Navigate to .