Skip to main content

Experiments Overview

The overview page will show you all your experiments by status. You can see which tests are running, scheduled, completed and cancelled. You may also get warning messages displayed when there aren’t enough users partaking in the test (when the sample size is too small).

This is a great place to keep an eye on all of your experiments and to track their overall Status. The Status bar will give you an insight into whether your experiment is running successfully based on whether it has enough users. Once an experiment is completed its status will change to Success and your results will be ready!

Experiments Overview

Here is a list of possible status messages on experiment and what they could mean:

  • “Scheduled for {Date of start_ts} at {Time of start_ts}” – SCHEDULED: the experiment is scheduled to start at a future date and time

  • “Roughly {days_left} days to go ({N} users acquired)” – RUNNING: the experiment is already acquiring new users and remote configs are being sent. Based on the already recruited players and how long the test has been running for, an estimation of how long the test should be running for until a winner can be determined is provided

  • “Warning, not getting enough users! ({N} users acquired)” – RUNNING: the experiment seems to have too few new users, and it may be that a winner can’t be determined due to small sample size

  • “Finalizing! {days_left} days to calculate retention” – RUNNING: the experiment already acquired enough users and ran for long enough, but it needs to wait for a few more days in order to calculate retention (giving newly acquired players time to return after chosen retention days)

  • “Success! Results are ready, but the test is still running” – COMPLETED: the experiment results are ready, however the remote configs are still being sent until you stop the test

  • “Stopped. Results are ready” – COMPLETED: success! your test results are ready and the experiment completed successfully

  • “Cancelled! ({N} users acquired)” – CANCELLED: the experiment was stopped before it met the criteria to run the winner model