### Hypothesis Test FAQ:

• Definition
• We have a null hypothesis $H_0$, and an alternative one $H_1$, we'll decide whether we should reject $H_0$ or not
• E.g. we are told a coin is fair and we would like to verify if it is so, so we have $H_0: p(h)=0.5$, and $H_1: p(h)\neq 0.5$. $p(h)$ is the probability getting head.
• We tossed the coin 10 times, and assume we have 4 heads. Assume coin is fair, what's the probability such events or even skewer events happen by chance?
• Let's do the math: p(at most 4 heads out of 10 tosses) = $\frac{1}{2^{10}}(C(10,1)+C(10,2)+C(10,3)+C(10,4)) = 0.37$ Apparently too large to reject.
• What about p(at most 3 heads out of 10 tosses) = $\frac{1}{2^{10}}(C(10,1)+C(10,2)+C(10,3)) = 0.17$ Still too large.
• What about p(at most 2 heads out of 10 tosses) = $\frac{1}{2^{10}}(C(10,1)+C(10,2)) = 0.05$ Then we are 95% sure this doesn't happen by chance and we can reject null at confidence level of 95%.

• Parameters
• Paired?
• Parametric or Non-parametric?
• Degree of freedom.
• Directional? Is it enough to be different? Or we can testing if it's significantly better(or worse)?

• Randomization Test
• Draw random samples...
• Chi-Square Test Example
• Establish the statistic: $\chi^2 = \frac{(O - E)^2}{E}$. $O$ is observed value. $E$ is expected value.
• Null hypothesis: coin is fair. Alternative hypothesis: coin is not fair
• Observation: tossed coin 10 times, and get 4 heads.
• Establish the test-statistic: $\frac{(4-5)^2+(6-5)^2}{5} = 0.4$. Degree of freedom is 1 since there are only two outcomes of a toss. By looking at the significance table (http://sites.stat.psu.edu/~mga/401/tables/Chi-square-table.pdf) we can't reject null.
• If we get 3 heads: $\frac{(3-5)^2+(7-5)^2}{5} = 1.6$ we can't reject.
• get 2 heads: $\frac{(2-5)^2+(8-5)^2}{5} = 3.6$ we reject at 95% confidence level.
• get 1 heads: $\frac{(1-5)^2+(9-5)^2}{5} = 6.4$ we reject at 99% confident level.
• How many we need in order to reject for a sample of 20? 4 heads or less to reach 95% confidence level
• T Test
• 1-sample, 2-sample, and paired t-test.
• 1-sample: compare one sample mean to null hypothesis.
• 2-sample: compare two sample means
• paired t-test: compare paired observation (before & after).
• Establish the test-statistic: $\frac{(4-5)^2+(6-5)^2}{5} = 0.4$. Degree of freedom is 1 since there are only two outcomes of a toss. By looking at the significance table (http://sites.stat.psu.edu/~mga/401/tables/Chi-square-table.pdf) we can't reject null.
• If we get 3 heads: $\frac{(3-5)^2+(7-5)^2}{5} = 1.6$ we can't reject.
• get 2 heads: $\frac{(2-5)^2+(8-5)^2}{5} = 3.6$ we reject at 95% confidence level.
• get 1 heads: $\frac{(1-5)^2+(9-5)^2}{5} = 6.4$ we reject at 99% confident level.
• How many we need in order to reject for a sample of 20? 4 heads or less to reach 95% confidence level
• Likelihood Ratio Test
• $D = -2\ln\frac{L(null model)}{L(alternative model)} = -2LL(null) - 2LL(alternative)$ and D is Chi-square distributed with df = #params of alternative model - #params of null model