Skip to main content

Best Practices & FAQs

Best Practices & Tips​

  • Consider when, where and how the test was run before applying results broadly: A/B test results reflect how users behaved under specific conditions during the test. If your game changes significantly after the test, or if player behavior changes in general over time, the results might no longer apply. Also, results from one region, platform, or build may not apply to others. For example, a test run on Android in the US might give different results than the same test on iOS in Europe.
  • Let tests run long enough: Ideally, until you reach statistical confidence for a Goal Metric.
  • Avoid overlapping tests: Users can only be in one test at a time, so consider how many new users you have coming into your game before running multiple tests. Multiple tests with a small number of new users will take longer to complete.
  • Segment results: Use filters to compare behavior by platform, country, or acquisition channel after the test is complete.

FAQs & Considerations​

Who gets added to a test? Based on your preference when creating a test either new users or both existing or new users are added to the test.

Can a user be in more than one test? No, users can only be enrolled in one active experiment at a time. In case multiple experiments are running, our server will randomly assign your player to only one of the experiments that they qualify for based on the test conditions.

Are offline users included? No. If the user is offline during their first session, they will not be assigned to any experiment.

How do A/B test configs interact with global remote configs? When a user is enrolled in an A/B test, any config keys defined in the test variant will take priority over the value in your global remote configs. This ensures that the user is always exposed to the correct variant configuration.

For example:

Global Config: enemy_speed = 3 A/B Variant A: enemy_speed = 2 A/B Variant B: enemy_speed = 1

A user assigned to Variant A will receive enemy_speed = 2, even though the global setting is 3.