I also teach elements of critical thinking in some of the actuarial normative skills workshops I run. By the time students get to me there, they are often already pessimistic and cynical when it comes to core areas of work (to be clear, this is a criticism of the profession, with a silver lining of critical thinking). However, there are plenty of other areas where their minds still seem susceptible to fake news.

This re-energised my interest in this area. I’ll blog on this topic more in future. One area I discovered in my research is the Illusory Truth Effective.

The Illusory Truth Effect is a really disappointing insight into how poorly our brains do at identifying truth. At its core, it says that when subjects are exposed to facts multiple times, even if the facts are highlighted as being false, still increases the probability that those subjects will view the facts as true. So whether it is on a Sunday morning surrounded by friends and family, or on the couch on your own infinitely browsing social media, what you see and hear and read becomes true in some proportion of minds.

It also applies to election campaigns and political rhetoric.

I’m going to test this with my actuarial students next chance I get.

This is a topic of direct personal and business relevance, but also of a technical forecasting and measurement perspective. Very little I’ve seen so far gives my confidence in the forecasting, which is either because of poor forecasting or from very limited communication.

I don’t know which bothers me more.

I’m going further than just restarting, I’m committing to a new blog post each day for October. Now the reasons for having stopped blogging haven’t suddenly changed, so it’s likely that some of these posts will be short. (And similarly, some of them long.) Since the decision was made last night, I also haven’t though through anything like a full plan for the month.  I invite you along to see how it goes.

I’m probably not alone in being slightly more jaded, slightly less optimistic than I was two years ago. A summary of the two years might make its way into another post, more to help me collect my thoughts than anything else.

Cape Town is experiencing an intense, multi-year drought and there is a real possibility of the city running out of water before next winter. I will definitely be blogging more about the vacuum of credible communication and forecasting on this front in a later post. For now, a single-purpose website http://www.howmanydaysofwaterdoescapetownhaveleft.co.za/  is currently proclaims (they update weekly, I think, based on updated weekly reports of dam levels) that we have 151 days of water left and will run out of useable water on 1 March 2018.

For now, the claims of cholera in Puerto Rico have not been proven, but it does feel like it’s only a matter of time. Anyone fretting over drinking water in Cape Town should probably bump diseases such as cholera up their list.

The official position of the City of Cape Town is still “we won’t run out of water”, but there are reasons to doubt this and be concerned. I’m keen to work out objectively what the level of risk is. To that end, it would have been useful to be able to dissect the http://www.howmanydaysofwaterdoescapetownhaveleft.co.za/  methodology to understand how credible their forecast is. This is the entire disclosure of their methodology:

Using our recent consumption as a model for future usage, we’re predicting that dam levels will reach 10% on the 1st of March, 2018.

I’m not losing sleep over their forecast. So for now, sleep.

The paper isn’t paygated so check it out – it’s only 6 pages so definitely accessible. Don’t worry about the couple of typos in the paper, bizarre as it may be to find them in a paper that presumably was reviewed, the ideas are still good.

The key idea is that prediction markets usually focus on binary events. Will Person Y win the election? Will China invade Taiwan? These outcomes are relatively easy to predict and circumvent important challenges of extreme outcomes and Taleb’s Black Swans.

A quote from the paper, itself quoting Taleb’s book, Fooled By Randomness, sums up the problem of trying to live in. Binary world when the real world has a wide range of outcomes.

In Fooled by Randomness, the narrator is asked “do you predict that the market is going up or down?† “Up†, he said, with confidence. Then the questioner got angry when he discovered that the narrator was short the market, i.e., would benefit from the market going down. The trader had a difficulty conveying the idea that someone could hold the belief that the market had a higher probability of going up, but that, should it go down, it would go down a lot. So the rational response was to be short.

A MODEL ISN’T REALITY

Any simplified model I can imagine will also therefore not match reality exactly. The closer the model gets to the real world in more scenarios, the better it is.

Not all model parameters are created equal

Part of the approach to getting a model to match reality as closely as possible is calibration. Models will typically have a range of parameters. Some will be well-established and can be set confidently without much debate. Others will have a range of reasonable or possible values based on empirical research or theory. Yet others will be relatively arbitrary or unobservable.

We don’t have to guess these values, even for the unobservable parameters. Through the process of calibration, the outputs of our model can be matched as closely as possible to actual historical values by changing the input parameters. The more certain we are of the parameters a priori the less we vary the parameters to calibrate the model. The parameters with most uncertainty are free to move as much as possible to fit the desired outputs.

During this process, the more structure or relationships that can be specified the better. The danger is that with relatively few data points (typically) and relatively many parameters (again typically) there will be multiple parameter sets that fit the data with possibly only very limited difference in “goodness of fit” for the results. The more information we add to the calibration process (additional raw data, more narrowly constrained parameters based on other research, tighter relationships between parameters) the more likely we are to derive a useful, sensible model that not only fits out calibration data well but also will be useful for predictions of the future or different decisions.

How not to calibrate a model

Scientific American has a naive article outlining “why economic models are always wrong”. I have two major problems with the story:

1. All models are wrong. Some are useful (George Box). “wrongness” isn’t a problem with a model, but lack of usefulness is. The headline demonstrates a starting point poorly informed about the point of economic models.
2. The calibration approach criticised in the article is an extremely poor way to calibrate a model. No serious researcher thinks that is the right way to calibrate a model. So the article merely creates a straw man and then demonstrates how easy it is to knock the argument over.

Calibration and back-testing on separate data sets

The right way to calibrate a model is to separate the data-set into at least two independent subsets. Firstly, the “training set” or portion from which we will calibrate our parameters to get them to match as closely as possible the data. Again, this should make use of all information available and may give rise to several competing models that appear to fit the data similarly well.

The next step is crucial. We back-test the derived models against the second subset of data. This data comes from the same reality (perhaps a different time period) as used to calibrate the model, but the model won’t trivially match the data because none of that data was used to calibrate the model in the first place.

The importance of back-testing

Back-testing is critically important in the model building process, but back-testing against the same data used to calibrate the model is worth than useless (since it takes time and effort and an create a false sense of accuracy or reliability in the model.) Separating the data into two or more subsets is absolutely required, although it has the unfortunate side-effect of reducing the size of the data-set available for calibration.

Yes, it really matters.

A common example of the dangers of bad models fitting data well is with Economic Scenario Generators. These simulate economic scenarios to be used in valuing complex financial securities. If a model is properly calibrated, it will recreate the observable market prices of a wide range of instruments. However, the model could be a black-box neural network, a carefully constructed theoretical model with plausible relationships and constraints, or the proverbial ten thousand (possibly inebriated) monkeys. If all three models are perfectly calibrated to observable market prices, is any of the models inferior to any of the others?

Clearly the answer is yes, but only when it comes to extrapolation. I have far more confidence in the model’s ability to create “market consistent” valuations for instruments that do not have observable prices in the market if I understand how the mechanics of the model make sense on a level other than pure calibration.

Trivial 3 point example

The example above shows a perfect of a quadratic and cubic model to 3 data points. From this graph, both models appear exactly the same.

However, if we use the model to extrapolate to future time periods, the results are very different. Without additional data to back-test the results on, it’s not possible to tell whether either or any of these models is appropriate, but clearly both can’t be correct.

It’s chock-full of analysis, numbers, tables and charts showing how as much as things change, the scope for financial crises changes very little.  The comparison of Developed and Emerging Markets is particularly interesting in that the differences, while they do exist, are far smaller than stereotypical views.  Emerging Markets do tend to have more ongoing sovereign defaults, but the frequency of banking crises is little different. Weirdly, some aspects of Emerging Market crises (such as employment impacts) are less than average for the Developed World.

It isn’t really the book’s fault, but this was one of the few books that I struggled with on my kindle – the graphs and charts and captions to figures were particularly difficult to read. Perhaps they would look better on the Kindle DX (the larger model) or even an iPad or something.

Although the book doesn’t focus on the current (still-happening, if you weren’t paying attention) financial crisis, there are several chapters dedicated to it with an analysis of the economic indicators leading up to the crash. Now it’s incredibly easy to predict an event after it’s happened, but I’m still hopeful that the results can be useful in predicting future problems and potentially impacting economic policies and regulations for the better.

Some key conclusions from the book for predictors of financial crises:

• markedly raising asset prices (yes, and in particular house prices given the likely co-factor of increases in debt levels)
• slowing real economic activity
• large current account deficits
• sustained debt build-ups (public and/or private)
• large and sustained capital inflows to a country
• financial sector liberalisation or innovation

That last point was particularly interesting for me – for all the statements that the US economy’s brilliant use of innovation and reduced regulations being a risk mitigant, history suggests this as a cause for the crisis.

For me, what was quite worrying is how well South Africa matches many of these points in the 2000s.  It seems that we either got off very lightly, or there is still an extended period of difficulty ahead.

After banking crises, house prices typically decline in real terms by 35.5% and this slump lasts on average 6 years. Now South Africa didn’t have a bank failure, so it may be that we missed the definition of “banking crisis”. However, given the pullback in credit offered along with international banking crises and property market declines, this suggests we’re in for an extended period of property market stagnation.

Steven Levitt (of Freakonomics fame) discussed an old paper of his and its usefulness in predicting US mid-term elections. This is now a 16 year-old model, which presumably could benefit with some updating for the last 16 years worth of data.

It does, currently anyway, give very similar answers to one of the biggest prediction markets operating, InTrade.com.

So, how does one not lose money with the Make a Million competition?

Don’t enter.

You are overwhelmingly like to lose money if you enter this competition. I’ve said this before, and I’ve been right before. I’m right again.

There’s also the little idea that the  structure of the Make a Million competition increases risks of  financial meltdown

Let’s look at some hard statistics to show what I mean.

Telling statistics (what they don’t show)

In the MaM presentation, the organisers include some interesting statistics about number of trades, trading activity and many other metrics.

They don’t show average returns or performance.

So let’s look at some of the numbers:

Raw return data (excluding prize money) based on 2009 MaM competition.

Suddenly the competition doesn’t look so great, does it?  (This isn’t the first time, here is my analysis of the Comedy and Tragedy that was the 2008 Make a Million competition.)

Here’s a little explanation of each of the items in the table:

Average Return

This is the return than you can expect to make on average. Yes, that’s a loss of over 10% of your investment. For all the talk about trading opportunities by the MaM organisers and sponsors, the trading result of this competition (ignoring prizes) is that more money is lost than is made.

Expected Loss

This is the total Rand amount you will lose on average (again ignoring prizes) by entering the competition. Quite a steep price. (The prize money makes the competition profitable on average, but only in a very skewed manner that only helps one person.  More on this in a bit)

Median Return

This is the return that half the entrants earned less than, and half the entrants earned more than. Half the participants lost more than 15% of their investment.

Mode Return

This is less intuitive to understand. The most common result for a entrant was to lose 9% of their starting stake.

Probability of breaking even, earning less than 10% or doubling money

Hopefully these are reasonably self-explanatory. What’s clear is that there is a high probability of doing badly, and a low probability of doing well.

Probability of winning

Ultimately your probability of winning serious money is still very low.

Great returns and manageable risk?

The MaM roadshow presentation concludes that there exist opportunities for great returns and manageable risk. I’m not sure what definition they’re using for “great returns” or “manageable risk” but in my book the returns are low and the risk is high. Look at the figures in the table above and explain how that can be interpreted any differently.

Is there any good news at all?

In fairness, there is a million rand prize available for the winner. This doesn’t change the probability of winning, the probability of earning 10%, the probability of doubling your money, the probability of losing any particular amount, but does add extra winnings to the best performer, which increases the average return considerably to 8.5% over the period. So yes, if you win, you will win. Remember the 0.2% probability of winning. In odds, that is about 500:1 against.

Winning strategies

There are some strategies than can help you win Make a Million.  They’re not really legal or ethical but you might want to know the sort of thing your competitors may be up to.

Their housing price index is more robust (and more independent) than those of the banks based off their own data and target markets. Rather than consider only the average price of houses sold in that particular month (which is a function of house price growth / decline but also how the type, condition, size and location of the houses sold that month differ from the prior month and year) they consider repeat sales where the same property has been bought and sold more than once.

This data is combined or “chain-linked” to provide a continuous measure of house price inflation over time.

The result of all of this data, best-in-class methodology and analysis? When Lightstone says “opportunities abound in local market” I actually listen. Since their business model is to sell information, I’m more likely to trust what they say.