Back
to leedsth.nhs.ukResearch & Development Website
Statistics Support
Sample sizes: Effect size
One of the critical decisions is what size of effect you are looking for. Too small, and your sample size will be very large; too large, and you might miss an important smaller effect.
Smaller effects need larger sizes: and the relation is quadratic, so to detect an effect half as big you need 4 times as many!
Sometimes people ask for a ‘clinically important difference' but what does this mean? Is a 1% increase in survival ‘clinically important'? It is if you are in the 1%.
Sometimes this phrase is interpreted to mean ‘what difference would you need to persuade you to change practice'
Remember also that it is often ‘events' (such as deaths) that matter in the study. A 10% reduction in mortality means a 1 % change in survival if the existing mortality rate is 10%.
Or you might look at effects which have been obtained in related studies and look to detect a similar effect.
Because variation is important as well, the effect is often expressed in terms of number of standard deviations.
Back to Main Stats Menu Back to Sample Size
Variability
The variablility in a population (usually measured by the standard deviation ) is an important factor in determining sample sizes.
If you are looking at things you measure (such as blood pressure) rather than things you count (such as deaths) you need to know what you expect it to be. Not surprisingly it is easier to find changes if the variability is small rather than large.
Unfortunately it is not always easy to estimate; you may get information from pilot studies, or previous related work.
It is important to estimate the correct variability. If you are doing repeat measurements on the same patients (for example a crossover trial) it is the 'within patient variation' that matters and it is typically less than the 'between patient variation' which you get from a population study.
Standard statistical methods such as the 't-test' assume that variability is the same in both groups.
If you are dealing with counts or proportions, the variability depends on the value.
Counts for instance often follow a 'Position distribution' and in that case
Variance (square of standard deviation) = mean.
Back to Main Stats Menu Back to Sample Size
Significance and Type I errors
Suppose we carry out a clinical trial of a drug for treating breast cancer. We randomise patients into 2 groups of 100 each. One group gets the drug, the other placebo.
After 3 years we count the number dead in each group. We would not expect exactly the same number in each group even if there were no real effect. There would be chance variation in the number of survivors.
So how big does the difference have to be to cause us to reject the ‘null hypothesis'? It is possible to calculate the probability that a difference as large or larger that that observed could arise if there were no real difference. This is called the ‘significance level' (also known as ‘p- value', or a ) of the result.
This is therefore the probability of falsely rejecting the null hypothesis when it is in fact true: ie the probability of making a Type I error.
The smaller the p- value the less likely the result is to have arisen by chance.
Another way of seeing it is to regard it as a measure of the evidence against the null hypothesis.
The lower the p-value, the stronger the evidence.
‘p = 0.05' is the weakest evidence which would generally cause us to reject the null hypothesis.
Power and type II errors
We want to reject the null hypothesis when it is in fact false and the power of a test is the probability that it will do so.
It is the probability of not making a type II error.
Naturally it is usually easier to detect big differences than small ones and power depends on the actual difference.
In planning a study we want to ensure that it has a good chance of detecting the effect we are looking for.
Typically the lowest power that one would plan for is 80%.
The probability of making a type II error is often labelled b .
‘Power' is often expressed as a percentage but a proportion is also used. Be sure you know which is meant!
| Back to Main Stats Menu | Back to Power & Significance I |