Skip to main content

Best Practices & FAQs

Best Practices & Tips​

  • Consider when, where and how the test was run before applying results broadly: A/B test results reflect how users behaved under specific conditions during the test. If your game changes significantly after the test, or if player behavior changes in general over time, the results might no longer apply. Also, results from one region, platform, or build may not apply to others. For example, a test run on Android in the US might give different results than the same test on iOS in Europe.
  • Let tests run long enough: Ideally, until you reach statistical confidence for a Goal Metric.
  • Avoid overlapping tests: Users can only be in one test at a time, so consider how many new users you have coming into your game before running multiple tests. Multiple tests with a small number of new users will take longer to complete.
  • Segment results: Use filters to compare behavior by platform, country, or acquisition channel after the test is complete.

Choosing between Backward-compatible and Enhanced versions​

  • Use the Backward-compatible version when your player base includes users on older SDK versions. This ensures all players can participate in the test.
  • Use the Enhanced version when you need JSON configuration values or values longer than 2,000 characters. Confirm your SDK version supports this before starting the test.
  • If you are unsure which SDK versions your players are on, check your build distribution data in SegmentIQ > A/B Testing before creating the test.

User limit recommendations​

  • For most tests, the default behavior (no manual user limit) is the right choice. The platform caps participation at 1,000,000 users automatically.
  • Set a manual user limit when you want to restrict the test to a smaller sample before rolling out changes broadly.
  • The minimum manual user limit is 2,000 to ensure statistical validity.

FAQs & Considerations​

Who gets added to a test?​

Based on your preference when creating a test either new users or both existing or new users are added to the test.

Can a user be in more than one test?​

No, users can only be enrolled in one active experiment at a time. In case multiple experiments are running, our server will randomly assign your player to only one of the experiments that they qualify for based on the test conditions.

Are offline users included?​

No. If the user is offline during their first session, they will not be assigned to any experiment.

How do A/B test configs interact with Remote Configs?​

When a user is enrolled in an A/B test, any config keys defined in the test variant will take priority over the value in your Remote Configs. This ensures that the user is always exposed to the correct variant configuration.

For example:

Global Config: enemy_speed = 3 A/B Variant A: enemy_speed = 2 A/B Variant B: enemy_speed = 1

A user assigned to Variant A will receive enemy_speed = 2, even though the global setting is 3.

Can I use JSON values with any SDK version?​

No. JSON values require the Enhanced version, which needs a compatible SDK. Check the SDKs Compatible with Enhanced version for the minimum version that supports JSON values in A/B tests.

What happens if a player's SDK does not support JSON values?​

Players on SDK versions that do not support JSON will not receive the test configuration correctly. Make sure your game is running a compatible SDK version before creating an Enhanced test with JSON values.

Is there still a user limit on A/B tests?​

Yes. All A/B tests are capped at 1,000,000 users. The difference is that you no longer need to set this manually. By default, tests run without a user limit and stop acquiring players at the 1,000,000 cap. You can optionally enable a manual limit between 2,000 and 1,000,000.