A/B and Multivariate Testing

📘

A/B Paywall Testing is available on our Team plan. Please contact us if you are interested in this feature.

📘

Multivariate Testing is available on our Business plan. Please contact us if you are interested in this feature.

Setting up an A/B Test

1. Find your Campaign

Find your campaign rule under the placement label. Or Create a Campaign

2. Create A/B Test 🧪

Click "Create A/B Test"

3. Select paywalls and allocation

Select 2 paywalls to test. To run a traditional A/B test, select 2 different paywalls where the 1st paywall is the control.

Set your traffic allocation. This determines what percentage of your app users will see each of the paywalls chosen. The 2 values must add up to 100%.

📘

Users are assigned to the A/B test by user ID. This means if a user has multiple devices but is signed in to the same account on all of them, all of their devices will be assigned to the same side of the test split. But if the user gets a new device and doesn't sign in, this device could be assigned to the other side of the split.

4. Publish ⚡️

Publish changes to set the the test live!

In the campaigns list, you will see a branch icon indicating a test is running.

🚧

Best Practice

Limit changes to the campaign or paywalls while the test is running. If you want to make large paywall changes, duplicate the paywall and campaigns to create a clean copy.

Ending an A/B Test

Once you have found a winner, create a new campaign rule with 100% allocation to the winner paywall. Prioritize the new rule higher than the old test and go live with the new rule. Then deactivate the completed test. That way your analytics are clean.

How to Test Both Sides of an A/B Test

Method 1 - Use multiple devices or simulators

The easiest way is to test your campaign on multiple devices or simulators. However, there is a risk that all your devices or simulators will be assigned to the same side of the split. The best way to get around this is to temporarily give your test a 50/50 allocation. In this case, the odds are good that, if you run the campaign on 2 or 3 devices, you will see both paywall variants.

Method 2 - Force the Traffic Allocation

If you don't have access to multiple devices or simulators or if you can't get Method 1 to work, try using 1 device and manually forcing the allocation to 100% for the paywall you want to test. Then when you have confirmed that paywall is working, deactivate the campaign, set the other paywall to 100%, and set the campaign to live again to test the other side.
Just don't forget to set your allocation back to your original values when you are done!

A/B Testing Best Practices

Run the Test until at least 100 users have seen the Paywalls
To get good results on paywall conversion, make sure you run it long enough for at least 100 users to see the paywalls. The time required to reach this number of users will depend on your active user volume and how often the campaign is being launched in your code.

Don't Change Paywalls
Once a campaign has gone live with an A/B test and has started collecting impressions for paywalls A and B, we don't recommend changing the selected paywalls. Instead, create a new campaign running paywall A against paywall C and run that test separately. That way, you can collect accurate results for test A and B.

Keep Paywall Edits Small
Try to avoid making edits to your selected paywalls while running an A/B test. If you need to fix a mistake, you can do so, but try to avoid making big changes such as copy changes, large changes to colors or images, or especially changing the available products. Instead, create another paywall with these edits and run a new test with this paywall.