A/B and Multivariate Testing
Last updated
Last updated
Follow these steps to .
Nami Best Practice
Add tags to AB test campaigns to help organize them
Nami Best Practice
Always create new campaign rules when running an AB test
Scroll down to Paywalls section and click "Create A/B Test"
Your campaign rule must be paused to add an AB test
Select 2 paywalls to test. To run a traditional A/B test, select 2 different paywalls where the 1st paywall is the control.
Set your traffic allocation. This determines what percentage of your app users will see each of the paywalls chosen. The 2 values must add up to 100%.
To setup a multivariate test, click 'Add Segment' to add another paywall. You can have up to 10 segments in a multivariate test. Again, all segment totals must add up to 100%;
Publish changes to set the the test live!
In the campaigns list, you will see a branch icon indicating a test is running.
Nami Best Practice
Limit changes to the campaign or paywalls while the test is running. If you want to make large paywall changes, duplicate the paywall and campaigns to create a clean copy.
Once you have found a winner, you will want to promote the winner as the new control.
Create a new campaign rule with the same Audience Filters as the AB test paywall and with 100% allocation to the winner paywall
Prioritize the new rule higher than the test rule and active the new rule.
Deactivate and archive the completed test.
Run the Test until at least 100 users have seen the Paywalls To get good results on paywall conversion, make sure you run it long enough for at least 100 users to see the paywalls. The time required to reach this number of users will depend on your active user volume and how often the campaign is being launched in your code.
Don't Change Paywalls Once a campaign has gone live with an A/B test and has started collecting impressions for paywalls A and B, we don't recommend changing the selected paywalls. Instead, create a new campaign running paywall A against paywall C and run that test separately. That way, you can collect accurate results for test A and B.
Keep Paywall Edits Small Try to avoid making edits to your selected paywalls while running an A/B test. If you need to fix a mistake, you can do so, but try to avoid making big changes such as copy changes, large changes to colors or images, or especially changing the available products. Instead, create another paywall with these edits and run a new test with this paywall.
