A/B Test Basics

When we want to decide whether to launch a new feature (new web UI, sorting algorithm, etc.), A/B tests are usually performed, and depending on the significance results, we then make a decision whether to make that launch or not.

We've seen before how the required sample sizes are calculated, now let's see how to calculate the z-score, or p-value in order to make a change.

Here we still make two scenarios

1. Two Sample z-test for Comparing Two Means

Assume we have two independent and normally distributed distributions, then

$z = \frac{\bar{x_1}-\bar{x_2}}{\sqrt{\frac{\sigma_1^2}{n_1}}+\sqrt{\frac{\sigma_2^2}{n_2}}}$

2. Two Sample z-test for Comparing Two Proportions

Assume we have two independent distributions that follow binomial distributions, then 

$z = \frac{\hat{p_1}-\hat{p_2}}{\sqrt{p_0(1-p_0)(\frac{1}{n_1}+\frac{1}{n_2})}}$ (pooled)

$z = \frac{\hat{p_1}-\hat{p_2}}{\sqrt{\frac{p_1(1-p_1)}{n_1}+\frac{p_2(1-p_2)}{n_2}}}$ (unpooled)

, where $p_0$ is the average of $p_1$ and $p_2$.

Now, after calculating the z-values are are able to make a conclusion. We want our p-value to be smaller than the significance level. Since the commonly used significant level is 5%, the minimum z-score required is 1.96 for two sided test and 1.64 for one-sided test.