Techniques (Advanced)

In order to determine whether the difference between variants is statistically significant, we use various statistical hypothesis testing techniques:

• If the goal metric is well understood – including the range of its nominal values, the frequency and scale of outliers etc, we use Bayesian Inference which enables us to perform hypothesis testing even for small sample sizes.

• When the number of measurements is high, we also use bootstrapping techniques after data normalization and outlier detection.

• When the sample size is high enough, bootstrapping and Bayesian inference lead to the same results, but bootstrapping is less resource intensive.