A/B Test Allocation

How Users are Allocated

Nami randomly assigns each user a hexadecimal user ID. Each user ID is then assigned to a position from 0-100 based on their ID. This position will be used to allocate their split.

Example: If a user's assigned split position is 40, they will be on the A side of a 50/50 AB test, and the B side of a 34/33/33 ABC test.

If the user has multiple devices but is signed into the same account on all of them, each device has the same Nami User ID and will be assigned to the same side of the test split.

But if the user gets a new device and doesn't sign in, this device could be assigned to another side of the split.

If a user signs in in the middle of a session, they could start seeing a different side of the test split.

How to Test Both Sides of an A/B Test

Method 1 - Use multiple devices or simulators

Use this method only if you do not login users with Nami

If your app does not login users with Nami, the easiest way is to test your campaign on multiple devices or simulators.

However, there is a risk that all your devices or simulators will be assigned to the same side of the split. The best way to get around this is to temporarily give your test a 50/50 allocation. In this case, the odds are good that, if you run the campaign on 2 or 3 devices, you will see both paywall variants.

Method 2 - Use multiple accounts

If your app signs in users with Nami, using multiple devices will not work to test if you are logged in as the same user on all those devices.

Instead, find 2 or 3 unique accounts. Do a fresh install and login as each account individually. You will likely find accounts that are assigned to both sides of the test split.

Method 3 - Force the traffic allocation

If you can't get the above methods to work, try using 1 device and manually forcing the allocation to 100% for the paywall you want to test. Then when you have confirmed that paywall is working, deactivate the campaign, set the other paywall to 100%, and set the campaign to live again to test the other side. ⚠️ Just don't forget to set your allocation back to your original values when you are done!

Frequently Asked Questions

Q: I am looking at Impressionsfor a 50/50 AB test and the number of impressions for each paywall is not equal. Why is that?

A: Nami randomly assigns split position to all users, but these users aren't guaranteed to see the paywall equally, since users have different user patterns in the app. However, as your test size grows, the number of impressions is likely to grow more and more equal.