A/B Test Paywalls inside a Campaign

📘

Minimum Code Version

Support for the latest version of the Nami campaigns including A/B testing is available in our Android and Apple SDKs in versions 3.0.0+.
Looking for older code versions? Check out the v1.0 docs

📘

A/B Paywall Testing is available on our paid plans. Please contact us if you are interested in this feature.

Setting up an A/B Test

1. Add Campaign ➕

Click "+ Add Campaign"

2. Name your Campaign

Give your Campaign a Name

📘

If you have multiple campaigns, set the Campaign Label to explicitly call a specific campaign. Otherwise, running the campaign code will launch the first created campaign.

3. Select Audience Rules

If you only want to show your campaign to certain app users, select rules from the audience filters. Learn more.

4. Create A/B Test 🧪

Click "Create A/B Test"

5. Select paywalls and allocation

Select 2 different paywalls to show to users in this test.

Set your traffic allocation. This determines what percentage of your app users will see each of the paywalls chosen. The 2 values must add up to 100%.

📘

Users are assigned to the A/B test by user ID. This means if a user has multiple devices but is signed in to the same account on all of them, all of their devices will be assigned to the same side of the test split. But if the user gets a new device and doesn't sign in, this device could be assigned to the other side of the split.

6. Go Live ⚡️

Save Campaign and Set Campaign Live
Then run the campaign code to see the test in app!

🚧

Once the campaign is set live, you cannot change the Label, change the attached paywalls, edit the paywall allocation, or remove the A/B test without deactivating the campaign.

How to Test Both Sides of an A/B Test

Method 1 - Use multiple devices or simulators

The easiest way is to test your campaign on multiple devices or simulators. However, there is a risk that all your devices or simulators will be assigned to the same side of the split. The best way to get around this is to temporarily give your test a 50/50 allocation. In this case, the odds are good that, if you run the campaign on 2 or 3 devices, you will see both paywall variants.

Method 2 - Force the Traffic Allocation

If you don't have access to multiple devices or simulators or if you can't get Method 1 to work, try using 1 device and manually forcing the allocation to 100% for the paywall you want to test. Then when you have confirmed that paywall is working, deactivate the campaign, set the other paywall to 100%, and set the campaign to live again to test the other side.
Just don't forget to set your allocation back to your original values when you are done!

A/B Testing Best Practices

Run the Test until at least 100 users have seen the Paywalls
To get good results on paywall conversion, make sure you run it long enough for at least 100 users to see the paywalls. The time required to reach this number of users will depend on your active user volume and how often the campaign is being launched in your code.

Don't Change Paywalls
Once a campaign has gone live with an A/B test and has started collecting impressions for paywalls A and B, we don't recommend changing the selected paywalls. Instead, create a new campaign running paywall A against paywall C and run that test separately. That way, you can collect accurate results for test A and B.

Keep Paywall Edits Small
Try to avoid making edits to your selected paywalls while running an A/B test. If you need to fix a mistake, you can do so, but try to avoid making big changes such as copy changes, large changes to colors or images, or especially changing the available products. Instead, create another paywall with these edits and run a new test with this paywall.