Manage A/B tests
View results
To view the test results, select the View button, that can be found next to each experiment in the experiment list.
This particular A/B test has successfully ran and has produced results, but based on status, different messages could appear.
Once an A/B test has been selected, at the top of the page you will be able to see the experiment’s configuration details along with its current status.
The Test Details, that is located at top of the can also be used to look at experiment details.
Once the button is clicked a popup will appear detailing the test ID, who created the test, build information, test user limit, etc… .
Scrolling down you will find the Model table, with results of the model once it has successfully run. In the model table, the metric is taken for the duration of the experiment.
The model only runs once enough players have been acquired in the experiment and a minimum amount of time has passed, and the information from the table corresponds to this. Data in this table will be static once the model has run.
For example, the winning variant for the experiment in the screenshot above is the Control group, due to having a higher playtime, compared to the experimental group, Variant 1.
In addition, you can also look at an extended list of metrics calculated during the A/B test, by analyzing the table.
The table allows for analyzing different metrics other than the goal metric and their changes based on different variants.
| Area | Metrics |
|---|---|
| Engagement | |
| Advertising | |
| Revenue | |
| Conversion | |
| Custom | Customize your tab with multiple metrics |
Change metric visualization
It’s possible to change the metric the graph is analyzing, once the experiment is active by accessing the test and using the metric picker.
Status messages
Depending on the state of the A/B test more status messages can be displayed near the experiment name in the list:
| Status | Definition |
|---|---|
| Scheduled for {Date of start_ts} at {Time of start_ts} | The experiment is scheduled to start at a future date and time |
| Roughly {days_left} days to go {(N) users acquired)} | The experiment is acquiring new users and remote configs are being sent. Based on the already recruited players and how long the test has been running for, an estimation of how long the test should be running for until a winner can be determined is provided |
| Warning, not getting enough users! {(N) users acquired} | The experiment seems to have too few new users, and it may be that a winner can’t be determined due to small sample size. |
| Success! Results are ready, but the test is still running | The experiment results are ready, however the remote configs are still being sent until you stop the test |
| Stopped. Results are ready | Test results are ready and the experiment completed successfully |
| Canceled! {(N) users acquired} | The experiment was stopped before it met the criteria to run the winner model |
Stop acquiring users
If there is ever a need to stop acquiring A/B test users, there are 2 options:
- On the page with the list of A/B tests, select the vertical ellipsis that is visible near each of the different ones, and choose the delete option
- On the results page
One reason to stop a current A/B test is to direct new user traffic to another experiment.
Users who had already been assigned to a variant by then will continue to receive variant configs in their game. The metric results table will also be updated as long as the experiment is active. Because we require time for the model to run and be successful, we recommend to only stop acquiring users after a few days.
Stop experiment
In case you need to stop an experiment, you can find the options to do so in the same locations as stopping to acquire users.
Keep in mind that stopping an experiment can have different outcomes depending on the state of the test:
- Stopping a scheduled test: the experiment will never commence and no users will be assigned to it. The results page will have no data. The experiment will move to Canceled state. From there you’ll have the option to Delete experiment.
- Stopping a running test : the experiment will stop acquiring any new users and the winning model will never run. The results page will show Goal metric Results and Core metrics data for the acquired users until that point, but no model results.
- Stopping a complete test : an experiment is considered complete when it has run for enough time and has acquired enough players for our statistical models to determine if there is a clear winner and, if so, which variant is the one.
However, this experiment will continue running (i.e., acquiring new users and sending the remote configs to experiment acquired players) until you decide to stop it, despite having generated the model already. Once you stop the test, the results will still appear under the experiment and you can come back to them as many times as desired.