A/B and Multivariate Testing
Last updated
Last updated
A/B Paywall Testing is available on our Team plan and up.
Multivariate Testing is available on our Business plan and up.
Find your campaign rule under the placement label. Or Create a Campaign
Click "Create A/B Test"
Select 2 paywalls to test. To run a traditional A/B test, select 2 different paywalls where the 1st paywall is the control.
Set your traffic allocation. This determines what percentage of your app users will see each of the paywalls chosen. The 2 values must add up to 100%.
Users are assigned to the A/B test by user ID. This means if a user has multiple devices but is signed in to the same account on all of them, all of their devices will be assigned to the same side of the test split. But if the user gets a new device and doesn't sign in, this device could be assigned to the other side of the split.
To setup a multivariate test, click 'Add Segment' to add another paywall. You can have up to 10 segments in a multivariate test. Again, all segment totals must add up to 100%;
Publish changes to set the the test live!
In the campaigns list, you will see a branch icon indicating a test is running.
Nami Best Practice
Limit changes to the campaign or paywalls while the test is running. If you want to make large paywall changes, duplicate the paywall and campaigns to create a clean copy.
Once you have found a winner, create a new campaign rule with 100% allocation to the winner paywall. Prioritize the new rule higher than the old test and go live with the new rule. Then deactivate the completed test. That way your analytics are clean.
Method 1 - Use multiple devices or simulators
Use this method only if you do not login users with Nami
If your app does not login users with Nami, the easiest way is to test your campaign on multiple devices or simulators.
However, there is a risk that all your devices or simulators will be assigned to the same side of the split. The best way to get around this is to temporarily give your test a 50/50 allocation. In this case, the odds are good that, if you run the campaign on 2 or 3 devices, you will see both paywall variants.
If your app signs in users with Nami, using multiple devices will not work to test if you are logged in as the same user on all those devices.
Instead, find 2 or 3 unique accounts. Do a fresh install and login as each account individually. You will likely find accounts that are assigned to both sides of the test split.
Run the Test until at least 100 users have seen the Paywalls To get good results on paywall conversion, make sure you run it long enough for at least 100 users to see the paywalls. The time required to reach this number of users will depend on your active user volume and how often the campaign is being launched in your code.
Don't Change Paywalls Once a campaign has gone live with an A/B test and has started collecting impressions for paywalls A and B, we don't recommend changing the selected paywalls. Instead, create a new campaign running paywall A against paywall C and run that test separately. That way, you can collect accurate results for test A and B.
Keep Paywall Edits Small Try to avoid making edits to your selected paywalls while running an A/B test. If you need to fix a mistake, you can do so, but try to avoid making big changes such as copy changes, large changes to colors or images, or especially changing the available products. Instead, create another paywall with these edits and run a new test with this paywall.
If you can't get the above methods to work, try using 1 device and manually forcing the allocation to 100% for the paywall you want to test. Then when you have confirmed that paywall is working, deactivate the campaign, set the other paywall to 100%, and set the campaign to live again to test the other side. Just don't forget to set your allocation back to your original values when you are done!
