One or two data samples and two hypotheses. Both hypotheses could be true.
One hypothesis is called null hypothesis, the other alternative hypothesis.
The null hypothesis states that, for example, a treatement has no effect on the patient.
The alternatie hypothesis states, that is has an effect.
The question is: which hypothesis is true?
One starts to assume the null hypothesis to be true.
Then, the probility (p) of the outcome of the data set for the changed environment is calculated.
If p is small, this is taken as evidence against the null hypothesis. This is called »rejecting the null hypothesis«.
On the other hand, if p is not small, there is no such evidence. This is called »failing to reject the null hypothesis«.
The second question then is: »when is p small«?
A usual value for this limit is 5% (p=0.05). But that value can and must be adjusted to data and risk.
Type I and Type II errors
Type I error = α error
Type II error = β error
actual situation
null hypothesis true false
-------------------------------------------------
accept correct decision Type II error
reject Type I error correct decision
The probability of wrongly accepting the null hypothesis is β (That is, if the null hypothesis is in actuality false).
The smaller β the greater the probability of commiting a Type I error (and rejecting the null hypothesis when it'd be correct).
Most statisticians work with α=0.05 and β=0.2
Power: 1-β (=0.8) This is used to calculate the sample sizes necessary to detect a specified difference when the error variance is known (or can be guessed at).