A/B and Multivariate Testing

A/B Paywall Testing is available on our Team plan and up.

Multivariate Testing is available on our Business plan and up.

Setting up an A/B Test

1. Create a new Campaign Rule 🔎

Follow these steps to Create a Campaign.

Nami Best Practice

Add tags to AB test campaigns to help organize them

Nami Best Practice

Always create new campaign rules when running an AB test

2. Create A/B Test 🧪

Scroll down to Paywalls section and click "Create A/B Test"

Your campaign rule must be paused to add an AB test

3. Select paywalls and allocation

Select 2 paywalls to test. To run a traditional A/B test, select 2 different paywalls where the 1st paywall is the control.

Set your traffic allocation. This determines what percentage of your app users will see each of the paywalls chosen. The 2 values must add up to 100%.

To setup a multivariate test, click 'Add Segment' to add another paywall. You can have up to 10 segments in a multivariate test. Again, all segment totals must add up to 100%;

4. Publish ⚡️

Publish changes to set the the test live!

In the campaigns list, you will see a branch icon indicating a test is running.

Nami Best Practice

Limit changes to the campaign or paywalls while the test is running. If you want to make large paywall changes, duplicate the paywall and campaigns to create a clean copy.

Ending an A/B Test

Selecting an A/B test winner

Once you have found a winner, you will want to promote the winner as the new control.

1. Create a new campaign rule with the same Audience Filters as the AB test paywall and with 100% allocation to the winner paywall

2. Prioritize the new rule higher than the test rule and active the new rule.

3. Deactivate and archive the completed test.

A/B Testing Best Practices

Run the Test until at least 100 users have seen the Paywalls To get good results on paywall conversion, make sure you run it long enough for at least 100 users to see the paywalls. The time required to reach this number of users will depend on your active user volume and how often the campaign is being launched in your code.

A/B Test Allocation

Don't Change Paywalls Once a campaign has gone live with an A/B test and has started collecting impressions for paywalls A and B, we don't recommend changing the selected paywalls. Instead, create a new campaign running paywall A against paywall C and run that test separately. That way, you can collect accurate results for test A and B.

Keep Paywall Edits Small Try to avoid making edits to your selected paywalls while running an A/B test. If you need to fix a mistake, you can do so, but try to avoid making big changes such as copy changes, large changes to colors or images, or especially changing the available products. Instead, create another paywall with these edits and run a new test with this paywall.