Category Archives: statistics

The virtual irrelevancy of population size to required sample size

Posted on by
Reply

Statistics and sampling are fundamental to almost all of our understanding of the world. The world is too big to measure directly. Measuring representative samples is a way to understand the entire picture.

Popular and academic literature are both full of examples of poor sample selection resulting in flawed conclusions about the population. Some of the most famous examples relied on sampling from telephone books (in the days when phone books still mattered and only relatively wealthy people had telephones) resulting in skewed samples.

This post is not about bias in sample selection but rather the simpler matter of sample sizes.

Population size is usually irrelevant to sample size

I’ve read too often the quote: “Your sample was only 60 people from a population of 100,000.  That’s not statistically relevant.”  Which is of course plain wrong and frustratingly wide-spread.

Required Sample Size is dictated by:

  • How accurate one needs the estimate to be
  • The standard deviation of the population
  • The homogeneity of the population

Only in exceptional circumstances does population size matter at all. To demonstrate this, consider the graph of the standard error of the mean estimate as the sample size increases for a population of 1,000 with a standard deviation of the members of the population of 25.

Standard Error as Sample Size increases for population of 1,000

Standard Error as Sample Size increases for population of 1,000

The standard error drops very quickly at first, then decreases very gradually thereafter even for a large sample of 100. Let’s see how this compares to a larger population of 10,000. Continue reading →

Birdy statistics

Posted on by
Reply

I’m not sure about this fascinating article on birds evolving to avoid cars in the US.

The story is that fewer cliff swallows are being killed on the roads AND those birds killed have longer than average wings. The argument here is that longer wings make for less agility, making the birds more likely to be killed by cars.

So far so good. But then:

The authors of the study found that over a 30 year period, annual cliff swallow roadkill has declined steadily from 20 birds per season in 1984 and 1985 to less than five birds per season during the last five years. Over the same period, traffic volumes remained the constant and the overall bird populations increased.

I am not an ornithologist or evolutionary expert, but I just can’t see how between 20 and 5 birds killed per season will create enough selection pressure to change the wingspan.

The original research summary is far more persuasive than the article. It shows graphs and statistical test results for decreasing average population wing size and increasing average road-kill wing size over time.

The explanation of why the average wingspan for cliff swallows killed be vehicles should increase is left unexplained. It does rather suggest potential measurement or confirmation bias from the research team – once the hypothesis starts looking interesting it would be very easy to unintentionally bias the measurements.  Measuring wingspan accurately to within a few millimetres is fraught with risks of subjective error.

Further, it looks like around 3 data points contribute significantly to the low p values of the tests and I would be very curious to know how robust the results were to removal of these influential points. It looks like the trends might remain, but without anything close to the significant suggested by the original research.

Finally, the clustering of wing measurement points in certain years suggests different levels of care and accuracy in measurement and potential “anchoring and adjustment bias”. It’s very hard to apply the same measurement protocols over 30 years.

So, fascinating research, interesting conclusion, but I’m left somehow unconvinced. It’s a pity the statistics applied weren’t a little more robust and the obvious criticisms weren’t addressed.

 

US Life Expectancy and the dangers of superficial analysis

Posted on by
Reply

Life Expectancy is going up. In general. But what really matters isn’t the general but the specifics. I know it’s hard to work through maths and actual calculations, but it doesn’t help if you run your analysis off slogans.

US Life Expectancy is going up. But not as much “at retirement” as it is up “at birth” because all the improvements in infant mortality are irrelevant at retirement. Similarly, mortality improvements aren’t the same for all income bands. The detail matters when it comes to trillions of dollars of social security.

More on gender and analytical problems

Posted on by
Reply

Here is another interesting story with a gender angle. A study shows that stockholders in companies with women in the Board achieved better returns than those without.
The obvious and likely correct point is that women add something valuable to the Board and is the company performs better. Diversity is a good thing in general, not least when it comes to considering complex issues with multiple stakeholders. It makes a good deal of sense to get this result.

Of course it’s not the only possible reason. It’s also not absolute proof that putting women onto a men-only Board would improve performance.

The problem is cause and effect. It might be that enlightened Boards add well-performing companies are more likely to add women to their Board. It might be that successful companies spend the time to get their Board composition right.

Finally, it might be stronger than the diversity argument. Women may simply be better at running companies than men. It’s a pity there are too few women-only boards to compare their performance to help answer the question whether women are better board members than men or is the benefit simply one of diversity. Interesting implications for other forms of diversity on Boards too.

Time and age and cohorts

Posted on by
Reply

More women than men get home loans in South Africa up to the age of 39 or so. This is fascinating in itself and needs further analysis.

Unfortunately, the analysis in the linked story misses a key point. 30 year old women and 50 year old women differ in that there is a 20 year age gap and there is a 20 year generation gap.

The 50 year old women were born in 1962 and grew up in the 60s, 70s and 80s, were educated in that period and influenced by social norms in that period and were given the opportunities that period provided.

Importantly, this means less educated, lower paid, larger families and less career oriented in their 30s than 30 year old women (born in 1982) are today. We can’t extrapolate from that study and compare the two age groups and say it’s a function of age. It’s likely to be heavily influenced by how different the world is.

That aside, I would not have guessed that more women under 40 would be successfully applying for home loans than men. (Not because they “shouldn’t” I just didn’t expect it based on the greater opportunities still available to men.)

I do wonder how the number of applicants compares, and what differences remain after controlling for education, province and many more others. If only I could get my hands on the data!

Lose a Million

Posted on by

The Make a Million competition, as I’ve mentioned before, is an awful idea. It doesn’t promote investing or even “normal” trading, but rather massive, speculative risk-taking trading because the prize for performing well is nothing and the prize for performing best is significant.

I’m continually disappointed that Moneyweb continues to partner with this distraction.

As I’ve done in the past, I’ve analysed very quickly some of the results of the most recent competition. As background to that, the basic rules are:

  1. Put up R20,000 of your own money
  2. Trade over three months in currencies, commodities single stock futures and some index trackers.
  3. Whoever has the most at the end wins a million rand
  4. Everyone keeps what is left of their initial “investment”

So let’s be clear, there are no long-term investment learnings here.

The winner did return 165.5% over 3 months, which is not an impressive performance even though it might look like it.  The point is, given the volatility of the investment universe available for the competition and the encouragement towards rampant risk-taking, it’s entirely pedestrian performance.  It’s very likely an individual’s performance will be good given the wide range of possible outcomes.

Let’s look at some other statistics

Average performance -18.4%
Annualised average performance -73.4%
Proportion making a profit 26%
Total amount won -R1 020 762
Standard Deviation of performance 48.0%
Annualised standard deviation 96%

These are not performance statistics of which to be proud. They are similar to the losses incurred in prior competitions.

So in short, the competition cost the entrants in total just over a million rand. Losing a million rand is a great way to Make a Million.