- Definition
- We have a null hypothesis $H_0$, and an alternative one $H_1$, we'll decide whether we should reject $H_0$ or not
- E.g. we are told a coin is fair and we would like to verify if it is so, so we have $H_0: p(h)=0.5$, and $H_1: p(h)\neq 0.5$. $p(h)$ is the probability getting head.
- We tossed the coin 10 times, and assume we have 4 heads. Assume coin is fair, what's the probability such events or even skewer events happen by chance?
- Let's do the math: p(at most 4 heads out of 10 tosses) = $\frac{1}{2^{10}}(C(10,1)+C(10,2)+C(10,3)+C(10,4)) = 0.37$ Apparently too large to reject.
- What about p(at most 3 heads out of 10 tosses) = $\frac{1}{2^{10}}(C(10,1)+C(10,2)+C(10,3)) = 0.17$ Still too large.
- What about p(at most 2 heads out of 10 tosses) = $\frac{1}{2^{10}}(C(10,1)+C(10,2)) = 0.05$ Then we are 95% sure this doesn't happen by chance and we can reject null at confidence level of 95%.
- Parameters
- Paired?
- Parametric or Non-parametric?
- Degree of freedom.
- Directional? Is it enough to be different? Or we can testing if it's significantly better(or worse)?
- Randomization Test
- Draw random samples...
- Chi-Square Test Example
- Establish the statistic: $\chi^2 = \frac{(O - E)^2}{E}$. $O$ is observed value. $E$ is expected value.
- Null hypothesis: coin is fair. Alternative hypothesis: coin is not fair
- Observation: tossed coin 10 times, and get 4 heads.
- Establish the test-statistic: $\frac{(4-5)^2+(6-5)^2}{5} = 0.4$. Degree of freedom is 1 since there are only two outcomes of a toss. By looking at the significance table (http://sites.stat.psu.edu/~mga/401/tables/Chi-square-table.pdf) we can't reject null.
- If we get 3 heads: $\frac{(3-5)^2+(7-5)^2}{5} = 1.6$ we can't reject.
- get 2 heads: $\frac{(2-5)^2+(8-5)^2}{5} = 3.6$ we reject at 95% confidence level.
- get 1 heads: $\frac{(1-5)^2+(9-5)^2}{5} = 6.4$ we reject at 99% confident level.
- How many we need in order to reject for a sample of 20? 4 heads or less to reach 95% confidence level
- T Test
- 1-sample, 2-sample, and paired t-test.
- 1-sample: compare one sample mean to null hypothesis.
- 2-sample: compare two sample means
- paired t-test: compare paired observation (before & after).
- Establish the test-statistic: $\frac{(4-5)^2+(6-5)^2}{5} = 0.4$. Degree of freedom is 1 since there are only two outcomes of a toss. By looking at the significance table (http://sites.stat.psu.edu/~mga/401/tables/Chi-square-table.pdf) we can't reject null.
- If we get 3 heads: $\frac{(3-5)^2+(7-5)^2}{5} = 1.6$ we can't reject.
- get 2 heads: $\frac{(2-5)^2+(8-5)^2}{5} = 3.6$ we reject at 95% confidence level.
- get 1 heads: $\frac{(1-5)^2+(9-5)^2}{5} = 6.4$ we reject at 99% confident level.
- How many we need in order to reject for a sample of 20? 4 heads or less to reach 95% confidence level
- Likelihood Ratio Test
- $D = -2\ln\frac{L(null model)}{L(alternative model)} = -2LL(null) - 2LL(alternative)$ and D is Chi-square distributed with df = #params of alternative model - #params of null model

Reading:- http://vassarstats.net/textbook/ all kinds of tests