The virtual irrelevancy of population size to required sample size

Statistics and sampling are fundamental to almost all of our understanding of the world. The world is too big to measure directly. Measuring representative samples is a way to understand the entire picture.

Popular and academic literature are both full of examples of poor sample selection resulting in flawed conclusions about the population. Some of the most famous examples relied on sampling from telephone books (in the days when phone books still mattered and only relatively wealthy people had telephones) resulting in skewed samples.

This post is not about bias in sample selection but rather the simpler matter of sample sizes.

Population size is usually irrelevant to sample size

I’ve read too often the quote: “Your sample was only 60 people from a population of 100,000.  That’s not statistically relevant.”  Which is of course plain wrong and frustratingly wide-spread.

Required Sample Size is dictated by:

  • How accurate one needs the estimate to be
  • The standard deviation of the population
  • The homogeneity of the population

Only in exceptional circumstances does population size matter at all. To demonstrate this, consider the graph of the standard error of the mean estimate as the sample size increases for a population of 1,000 with a standard deviation of the members of the population of 25.

Standard Error as Sample Size increases for population of 1,000
Standard Error as Sample Size increases for population of 1,000

The standard error drops very quickly at first, then decreases very gradually thereafter even for a large sample of 100. Let’s see how this compares to a larger population of 10,000. Continue reading “The virtual irrelevancy of population size to required sample size”

XKCD and statisticians

I was looking for an XKCD comic to insert into a presentation on insurance regulations. Yes, I do that. I came across a comic I hadn’t seen before and that also illustrates a fundamentally important point.


Birdy statistics

I’m not sure about this fascinating article on birds evolving to avoid cars in the US.

The story is that fewer cliff swallows are being killed on the roads AND those birds killed have longer than average wings. The argument here is that longer wings make for less agility, making the birds more likely to be killed by cars.

So far so good. But then:

The authors of the study found that over a 30 year period, annual cliff swallow roadkill has declined steadily from 20 birds per season in 1984 and 1985 to less than five birds per season during the last five years. Over the same period, traffic volumes remained the constant and the overall bird populations increased.

I am not an ornithologist or evolutionary expert, but I just can’t see how between 20 and 5 birds killed per season will create enough selection pressure to change the wingspan.

The original research summary is far more persuasive than the article. It shows graphs and statistical test results for decreasing average population wing size and increasing average road-kill wing size over time.

The explanation of why the average wingspan for cliff swallows killed be vehicles should increase is left unexplained. It does rather suggest potential measurement or confirmation bias from the research team – once the hypothesis starts looking interesting it would be very easy to unintentionally bias the measurements.  Measuring wingspan accurately to within a few millimetres is fraught with risks of subjective error.

Further, it looks like around 3 data points contribute significantly to the low p values of the tests and I would be very curious to know how robust the results were to removal of these influential points. It looks like the trends might remain, but without anything close to the significant suggested by the original research.

Finally, the clustering of wing measurement points in certain years suggests different levels of care and accuracy in measurement and potential “anchoring and adjustment bias”. It’s very hard to apply the same measurement protocols over 30 years.

So, fascinating research, interesting conclusion, but I’m left somehow unconvinced. It’s a pity the statistics applied weren’t a little more robust and the obvious criticisms weren’t addressed.


US Life Expectancy and the dangers of superficial analysis

Life Expectancy is going up. In general. But what really matters isn’t the general but the specifics. I know it’s hard to work through maths and actual calculations, but it doesn’t help if you run your analysis off slogans.

US Life Expectancy is going up. But not as much “at retirement” as it is up “at birth” because all the improvements in infant mortality are irrelevant at retirement. Similarly, mortality improvements aren’t the same for all income bands. The detail matters when it comes to trillions of dollars of social security.

More on gender and analytical problems

Here is another interesting story with a gender angle. A study shows that stockholders in companies with women in the Board achieved better returns than those without.
The obvious and likely correct point is that women add something valuable to the Board and is the company performs better. Diversity is a good thing in general, not least when it comes to considering complex issues with multiple stakeholders. It makes a good deal of sense to get this result.

Of course it’s not the only possible reason. It’s also not absolute proof that putting women onto a men-only Board would improve performance.

The problem is cause and effect. It might be that enlightened Boards add well-performing companies are more likely to add women to their Board. It might be that successful companies spend the time to get their Board composition right.

Finally, it might be stronger than the diversity argument. Women may simply be better at running companies than men. It’s a pity there are too few women-only boards to compare their performance to help answer the question whether women are better board members than men or is the benefit simply one of diversity. Interesting implications for other forms of diversity on Boards too.

Time and age and cohorts

More women than men get home loans in South Africa up to the age of 39 or so. This is fascinating in itself and needs further analysis.

Unfortunately, the analysis in the linked story misses a key point. 30 year old women and 50 year old women differ in that there is a 20 year age gap and there is a 20 year generation gap.

The 50 year old women were born in 1962 and grew up in the 60s, 70s and 80s, were educated in that period and influenced by social norms in that period and were given the opportunities that period provided.

Importantly, this means less educated, lower paid, larger families and less career oriented in their 30s than 30 year old women (born in 1982) are today. We can’t extrapolate from that study and compare the two age groups and say it’s a function of age. It’s likely to be heavily influenced by how different the world is.

That aside, I would not have guessed that more women under 40 would be successfully applying for home loans than men. (Not because they “shouldn’t” I just didn’t expect it based on the greater opportunities still available to men.)

I do wonder how the number of applicants compares, and what differences remain after controlling for education, province and many more others. If only I could get my hands on the data!