Statistics and sampling are fundamental to almost all of our understanding of the world. The world is too big to measure directly. Measuring representative samples is a way to understand the entire picture.
Popular and academic literature are both full of examples of poor sample selection resulting in flawed conclusions about the population. Some of the most famous examples relied on sampling from telephone books (in the days when phone books still mattered and only relatively wealthy people had telephones) resulting in skewed samples.
This post is not about bias in sample selection but rather the simpler matter of sample sizes.
Population size is usually irrelevant to sample size
I’ve read too often the quote: “Your sample was only 60 people from a population of 100,000. That’s not statistically relevant.” Which is of course plain wrong and frustratingly wide-spread.
Required Sample Size is dictated by:
- How accurate one needs the estimate to be
- The standard deviation of the population
- The homogeneity of the population
Only in exceptional circumstances does population size matter at all. To demonstrate this, consider the graph of the standard error of the mean estimate as the sample size increases for a population of 1,000 with a standard deviation of the members of the population of 25.
The standard error drops very quickly at first, then decreases very gradually thereafter even for a large sample of 100. Let’s see how this compares to a larger population of 10,000.
It’s not an error that those graphs are almost identical. (see next section for why they’re not exactly the same.) Population size is irrelevant.
Finite Population Correction – or when population size begins to matter
Population size is usually irrelevant to sample size. There is an exception. Few people even know about this because it is so rarely relevant.
If one samples the entire population, there is no room for error in estimating statistics related to the population. The mean of the population is then known precisely. We don’t have a “sample” any more. The standard error of our estimate is thus zero since it is no longer an estimate but a direct measurement of the population statistics.
As the sample size approaches the population size, the sample error begins to decline. We adjust the normal standard error estimate by multiplying it by sqrt((Populaton – Sample) / (Population – 1)). In this case, you can see that the 100 population sample error does begin to decrease in the graph above.
You can go ahead and forget that formula now as you’ll probably never need it. It does explain the slight difference in the graphs in the previous section though.
Non-homogenous populations aka sample size still isn’t the answer
One last point – if the population is not homoegenous, this will typically increase the total population variability and result in higher standard errors. It adds risk in using a small sample size. The best solution is not to simply increase the population size. Cluster Sampling is a straightforward way of reducing the variability of overall estimates for population statistics and better understanding differences between the clusters.
So what determines the required sample size?
I get asked “how big a sample will we need” typically with only the population size to go on. Hopefully I’ve explained why the population size is irrelevant in almost all scenarios. How should you estimate the required sample size?
Ideally, you need an initial estimate of population standard deviation. You can get this from prior studies of similar populations or by doing a quick, small initial survey to estimate the standard error to set a sample size. You could also continue to sample under the standard error of your estimate became sufficiently low, but this often isn’t practical and can encourage unintended biases in terms of how the sample is increased over time.
Without anything to go on, I use the rule of thumb that 10 observations is usually the minimum number within a homogenous population to measure a single characteristic. 20 is better and usually, more doesn’t make much of a difference.
Smaller than you thought?