1 Proportion

Experiment: test color color of a button

• Click through probability: N(users who clicked) / N(total users)
• 1000 users in both control and treatment groups

Results:

• Control group: 1.1% CTP
• Treatment group: 2.3% CTP

Significance:

• Practical significant boundary: 0.01
• Significance level $\alpha$ : 0.05

Make a decision:

• Significant difference? Launch the “feature”?

Questions

1. Which hypothesis test to use?
2. What is the null hypothesis?
3. Is the result statistically significant?
4. Is the result practically significant?

• Bernoulli population: either clicks or doesn’t click
• Control group: n*p = 1000 * 1.1% = 11
• Treatment group: n * p = 1000 * 2.3% = 23
• Both np and n(1-p) are larger than 10, so we can consider it as large samples. Test statistic follows Z-distribution.

T-Test Z-Test The difference between ？

• https://zhuanlan.zhihu.com/p/120181558

Measurements

• Users clicked $X_{ct}$, $X_{tr}$
• Total number of users $n_{ct}$, $n_{tr}$

$P_{ct}$ = $X_{ct}$ / $n_{ct}$ = 11 / 1000
$P_{tr}$ = $X_{tr}$ / $n_{tr}$ = 23 / 1000

What is the null hypothesis?

We want to measure the difference of $P_{tr}$ and $P_{ct}$ .

d = $P_{tr}$ - $P_{ct}$

Null hypothesis:

$H_0$: $P_{tr}$ = $P_{ct}$ , d = 0
d ~ N(0, $S{E}^2$)

We don’t know the standard deviation of d, so we need to estimate it.

Test statistic:

TS = $(P_{tr}$ - $P_{ct})/SE$

Estimate a standard error:

• Choose a SE can represent both groups
• “Pooled” standard error

Compute “pooled” SE

• “Pooled” probability of a click, p’
• Total probability across 2 groups:

$P^{\prime }$ = $(X_{ct}+X_{tr})/(n_{ct}+n_{tr})$ = (11+23) / (1000+1000) = 0.017
Test statistics

TS = $(P_{tr}$ - $P_{ct})/SE$ = 0.012 / 0.00578 = 2.076

Is result statistically significant?

• critical z-score ($\alpha$: 0.05) = 1.96
• TS > 1.96 or TS < -1.96, reject null hypothesis
• In this example, Test is statistically significant.

Is result practically significant?

• Confidence interval of d

• Center of C.I. = 0.012 (This is $P_{tr}$ - $P_{ct}$ )

• Width of C.I. (margin of error)

m = Z * $S_{pool}$ = 1.96 * 0.00578 = 0.0113

CI of d: 0.012 ± 0.0113 = 0.0007 ～ 0.0233

Best guess: There is a practical significant change.
It’s possible the change is not practical significant.

Make launch decision:

• Not confident the change is practically significant.
• Not recommend launch the feature.

Checking statistical significance:

• Check if CI overlaps with 0: If it does, result is not statistically significant.
• Equivalent to comparing TS with critical value.

2 Mean

Experiment: if a new feature changes avg. number of posts

Correction: Mean of treatment is 1.7

What conclusion can you draw?

• Assume variances are similar.

Significance:

• Practical significant boundary: 0.05
• Significance level $\alpha$ : 0.05

Correction: Spool = 1.06

SS: Sum of square

Margin of error would be t-score*Spool (1/(1/nc +1/nt)^1/2), which would come to be ~0.51 which +/- from d-hat (0.6) would be above the significance level of 0.05

3 Welch’s T-test

Reference: https://www.youtube.com/watch?v=6uw0A3aKwMc