Category Archives: predictive modelling

How not to calibrate a model

Posted on by

Any model is a simplification of reality. If it isn’t, then it isn’t a model as rather is the reality.

A MODEL ISN’T REALITY

Any simplified model I can imagine will also therefore not match reality exactly. The closer the model gets to the real world in more scenarios, the better it is.

Not all model parameters are created equal

Part of the approach to getting a model to match reality as closely as possible is calibration. Models will typically have a range of parameters. Some will be well-established and can be set confidently without much debate. Others will have a range of reasonable or possible values based on empirical research or theory. Yet others will be relatively arbitrary or unobservable.

We don’t have to guess these values, even for the unobservable parameters. Through the process of calibration, the outputs of our model can be matched as closely as possible to actual historical values by changing the input parameters. The more certain we are of the parameters a priori the less we vary the parameters to calibrate the model. The parameters with most uncertainty are free to move as much as possible to fit the desired outputs.

During this process, the more structure or relationships that can be specified the better. The danger is that with relatively few data points (typically) and relatively many parameters (again typically) there will be multiple parameter sets that fit the data with possibly only very limited difference in “goodness of fit” for the results. The more information we add to the calibration process (additional raw data, more narrowly constrained parameters based on other research, tighter relationships between parameters) the more likely we are to derive a useful, sensible model that not only fits out calibration data well but also will be useful for predictions of the future or different decisions.

How not to calibrate a model

Scientific American has a naive article outlining “why economic models are always wrong”. I have two major problems with the story: Continue reading →

Book Review: This Time is Different

Posted on by

This Time is Different is a fascinating look at 8 centuries of financial crises including banking, currency and sovereign default.

It’s chock-full of analysis, numbers, tables and charts showing how as much as things change, the scope for financial crises changes very little.  The comparison of Developed and Emerging Markets is particularly interesting in that the differences, while they do exist, are far smaller than stereotypical views.  Emerging Markets do tend to have more ongoing sovereign defaults, but the frequency of banking crises is little different. Weirdly, some aspects of Emerging Market crises (such as employment impacts) are less than average for the Developed World.

It isn’t really the book’s fault, but this was one of the few books that I struggled with on my kindle – the graphs and charts and captions to figures were particularly difficult to read. Perhaps they would look better on the Kindle DX (the larger model) or even an iPad or something.

Although the book doesn’t focus on the current (still-happening, if you weren’t paying attention) financial crisis, there are several chapters dedicated to it with an analysis of the economic indicators leading up to the crash. Now it’s incredibly easy to predict an event after it’s happened, but I’m still hopeful that the results can be useful in predicting future problems and potentially impacting economic policies and regulations for the better.

Some key conclusions from the book for predictors of financial crises:

  • markedly raising asset prices (yes, and in particular house prices given the likely co-factor of increases in debt levels)
  • slowing real economic activity
  • large current account deficits
  • sustained debt build-ups (public and/or private)
  • large and sustained capital inflows to a country
  • financial sector liberalisation or innovation Continue reading →

Prediction: models versus market

Posted on by

This is not the best way to start serious analysis of models versus markets in the prediction space, but given that I’m writing an exam tomorrow I thought I should put the links out there now.  I’ll address this topic again in the future.

Steven Levitt (of Freakonomics fame) discussed an old paper of his and its usefulness in predicting US mid-term elections. This is now a 16 year-old model, which presumably could benefit with some updating for the last 16 years worth of data.

It does, currently anyway, give very similar answers to one of the biggest prediction markets operating, InTrade.com.

How not to lose money in Make a Million

Posted on by

I have a clear strategy for how not to lose money playing the Make a Million competition. As I explain it, you may come up with some smart tactics to win the competition and enhance your returns, but you’re on you’re own there.

So, how does one not lose money with the Make a Million competition?

Don’t enter.


You are overwhelmingly like to lose money if you enter this competition. I’ve said this before, and I’ve been right before. I’m right again.

There’s also the little idea that the  structure of the Make a Million competition increases risks of  financial meltdown

Let’s look at some hard statistics to show what I mean.

Telling statistics (what they don’t show)

In the MaM presentation, the organisers include some interesting statistics about number of trades, trading activity and many other metrics.

They don’t show average returns or performance.

So let’s look at some of the numbers:

Raw return data (excluding prize money) based on 2009 MaM competition.

Average Return -11.49%
Expected Loss R 1,149
Median Return -15.06%
Mode Return -9.12%
Probability of breaking even 25.00%
Probability of earning less than 10% 83.00%
Probability of doubling money 1.78%
Probability of winning 0.20%

Suddenly the competition doesn’t look so great, does it?  (This isn’t the first time, here is my analysis of the Comedy and Tragedy that was the 2008 Make a Million competition.) Continue reading →

Repo down by 50bps

Posted on by

Looks like my money is safe – Reserve Bank cut rates as predicted. Thinking about trying to predict for each MPC meeting then tracking my performance over time so I can be held accountable. Will mull over this first I am not that sure I’ll be sufficiently confident to stick my neck out in future!

Property investment – the value of data over opinions

Posted on by

Lightstone have a trick up their sleeves. Their raison d’être is collecting, analysing, understanding and packaging data for themselves and others to use to understand past, current and future property valuations.

Their housing price index is more robust (and more independent) than those of the banks based off their own data and target markets. Rather than consider only the average price of houses sold in that particular month (which is a function of house price growth / decline but also how the type, condition, size and location of the houses sold that month differ from the prior month and year) they consider repeat sales where the same property has been bought and sold more than once.

This data is combined or “chain-linked” to provide a continuous measure of house price inflation over time.

House Price Inflation 2010

House Price Inflation 2010 source: lightstone.co.za

The result of all of this data, best-in-class methodology and analysis? When Lightstone says “opportunities abound in local market” I actually listen. Since their business model is to sell information, I’m more likely to trust what they say.

Most decisions are made without all the information

Posted on by

Tyler Reed blogs about entrepreneurs having to make decisions with limited information.

It’s almost all unknown

I don’t disagree.  It’s just that almost every meaningful decision ever made is made without all the information.

Unknowns can be categorised a hundred different ways. One way is to think about:

  1. Unknown past information
  2. Uncertainty around the current situation or position
  3. Unknown future outcomes

Even a game like chess, where the past history of the game is easily known by good players, the current position is clearly visible and all the possible moves are knowable, it is not possible have all the information about how your opponent will react to your move.

How to deal with decision making under uncertainty – part 1

Tyler suggests that gut-based decision making can be effective much of the time – and it can. It there genuinely is no time for anything more than an instinctive reaction, you probably are best going with your gut.

Even if you have plenty of time, listening to your guy to formulate an idea is a great idea. Insight comes partly from experience and the reinforced neural pathways of our learning brain. If you stop with the gut though, you are missing out. There is a tremendous amount of research showing how ridiculously badly our instincts perform in many areas, particularly those relating to uncertainty and complexity! Continue reading →

CPI at 3.7% for July 2010

Posted on by

From Stats SA

The headline inflation rate in July 2010 (i.e. the Consumer Price Index for all urban areas in July 2010 compared with that at July 2009) was 3,7%

The official inflation rate (i.e. the percentage change in the CPI for all urban areas in July 2010 compared with that in July 2009) was 3,7% at July 2010. This rate was 0,5 of a percentage point lower than the corresponding annual rate of 4,2% in June 2010 (i.e. the Consumer Price Index for all urban areas in June 2010 compared with that in June 2009).

From June 2010 to July 2010 the Consumer Price Index for all urban increased by 0,6%

CPI Headline July 2010 = 3,7%

So this is close to the bottom of our 3% to 6% inflation targeting range. Economic growth is struggling, unemployment is high, but we haven’t reduced interest rates? Something here is a little odd.

I’ll put another $100 in Kiva, to be “microlent” to businesses and people across the world, if the next monetary policy committee meeting doesn’t cut interest rates.