A/B Test Allocation
How Users are Allocated
Nami randomly assigns each user a hexadecimal user ID. Each user ID is then assigned to a position from 0-100 based on their ID. This position will be used to allocate their split.
Example: If a user's assigned split position is 40, they will be on the A side of a 50/50 AB test, and the B side of a 34/33/33 ABC test.
If the user has multiple devices but is signed into the same account on all of them, each device has the same Nami User ID and will be assigned to the same side of the test split.
But if the user gets a new device and doesn't sign in, this device could be assigned to another side of the split.
If a user signs in in the middle of a session, they could start seeing a different side of the test split.
How to Test Both Sides of an A/B Test
Method 1 - Use multiple devices or simulators
Use this method only if you do not login users with Nami
If your app does not login users with Nami, the easiest way is to test your campaign on multiple devices or simulators.
However, there is a risk that all your devices or simulators will be assigned to the same side of the split. The best way to get around this is to temporarily give your test a 50/50 allocation. In this case, the odds are good that, if you run the campaign on 2 or 3 devices, you will see both paywall variants.
Method 2 - Use multiple accounts
If your app signs in users with Nami, using multiple devices will not work to test if you are logged in as the same user on all those devices.
Instead, find 2 or 3 unique accounts. Do a fresh install and login as each account individually. You will likely find accounts that are assigned to both sides of the test split.
Method 3 - Force the traffic allocation
Frequently Asked Questions
Q: I am looking at Impressionsfor a 50/50 AB test and the number of impressions for each paywall is not equal. Why is that?
A: Nami randomly assigns split position to all users, but these users aren't guaranteed to see the paywall equally, since users have different user patterns in the app. However, as your test size grows, the number of impressions is likely to grow more and more equal.
Last updated