A MODEL ISN’T REALITY

Any simplified model I can imagine will also therefore not match reality exactly. The closer the model gets to the real world in more scenarios, the better it is.

Not all model parameters are created equal

Part of the approach to getting a model to match reality as closely as possible is calibration. Models will typically have a range of parameters. Some will be well-established and can be set confidently without much debate. Others will have a range of reasonable or possible values based on empirical research or theory. Yet others will be relatively arbitrary or unobservable.

We don’t have to guess these values, even for the unobservable parameters. Through the process of calibration, the outputs of our model can be matched as closely as possible to actual historical values by changing the input parameters. The more certain we are of the parameters a priori the less we vary the parameters to calibrate the model. The parameters with most uncertainty are free to move as much as possible to fit the desired outputs.

During this process, the more structure or relationships that can be specified the better. The danger is that with relatively few data points (typically) and relatively many parameters (again typically) there will be multiple parameter sets that fit the data with possibly only very limited difference in “goodness of fit” for the results. The more information we add to the calibration process (additional raw data, more narrowly constrained parameters based on other research, tighter relationships between parameters) the more likely we are to derive a useful, sensible model that not only fits out calibration data well but also will be useful for predictions of the future or different decisions.

How not to calibrate a model

Scientific American has a naive article outlining “why economic models are always wrong”. I have two major problems with the story:

1. All models are wrong. Some are useful (George Box). “wrongness” isn’t a problem with a model, but lack of usefulness is. The headline demonstrates a starting point poorly informed about the point of economic models.
2. The calibration approach criticised in the article is an extremely poor way to calibrate a model. No serious researcher thinks that is the right way to calibrate a model. So the article merely creates a straw man and then demonstrates how easy it is to knock the argument over.

Calibration and back-testing on separate data sets

The right way to calibrate a model is to separate the data-set into at least two independent subsets. Firstly, the “training set” or portion from which we will calibrate our parameters to get them to match as closely as possible the data. Again, this should make use of all information available and may give rise to several competing models that appear to fit the data similarly well.

The next step is crucial. We back-test the derived models against the second subset of data. This data comes from the same reality (perhaps a different time period) as used to calibrate the model, but the model won’t trivially match the data because none of that data was used to calibrate the model in the first place.

The importance of back-testing

Back-testing is critically important in the model building process, but back-testing against the same data used to calibrate the model is worth than useless (since it takes time and effort and an create a false sense of accuracy or reliability in the model.) Separating the data into two or more subsets is absolutely required, although it has the unfortunate side-effect of reducing the size of the data-set available for calibration.

Yes, it really matters.

A common example of the dangers of bad models fitting data well is with Economic Scenario Generators. These simulate economic scenarios to be used in valuing complex financial securities. If a model is properly calibrated, it will recreate the observable market prices of a wide range of instruments. However, the model could be a black-box neural network, a carefully constructed theoretical model with plausible relationships and constraints, or the proverbial ten thousand (possibly inebriated) monkeys. If all three models are perfectly calibrated to observable market prices, is any of the models inferior to any of the others?

Clearly the answer is yes, but only when it comes to extrapolation. I have far more confidence in the model’s ability to create “market consistent” valuations for instruments that do not have observable prices in the market if I understand how the mechanics of the model make sense on a level other than pure calibration.

Trivial 3 point example

Perfect fit of two models to 3 data points

The example above shows a perfect of a quadratic and cubic model to 3 data points. From this graph, both models appear exactly the same.

However, if we use the model to extrapolate to future time periods, the results are very different. Without additional data to back-test the results on, it’s not possible to tell whether either or any of these models is appropriate, but clearly both can’t be correct.

Extrapolation of the two models shows divergent results

It’s chock-full of analysis, numbers, tables and charts showing how as much as things change, the scope for financial crises changes very little.  The comparison of Developed and Emerging Markets is particularly interesting in that the differences, while they do exist, are far smaller than stereotypical views.  Emerging Markets do tend to have more ongoing sovereign defaults, but the frequency of banking crises is little different. Weirdly, some aspects of Emerging Market crises (such as employment impacts) are less than average for the Developed World.

It isn’t really the book’s fault, but this was one of the few books that I struggled with on my kindle – the graphs and charts and captions to figures were particularly difficult to read. Perhaps they would look better on the Kindle DX (the larger model) or even an iPad or something.

Although the book doesn’t focus on the current (still-happening, if you weren’t paying attention) financial crisis, there are several chapters dedicated to it with an analysis of the economic indicators leading up to the crash. Now it’s incredibly easy to predict an event after it’s happened, but I’m still hopeful that the results can be useful in predicting future problems and potentially impacting economic policies and regulations for the better.

Some key conclusions from the book for predictors of financial crises:

• markedly raising asset prices (yes, and in particular house prices given the likely co-factor of increases in debt levels)
• slowing real economic activity
• large current account deficits
• sustained debt build-ups (public and/or private)
• large and sustained capital inflows to a country
• financial sector liberalisation or innovation

That last point was particularly interesting for me – for all the statements that the US economy’s brilliant use of innovation and reduced regulations being a risk mitigant, history suggests this as a cause for the crisis.

For me, what was quite worrying is how well South Africa matches many of these points in the 2000s.  It seems that we either got off very lightly, or there is still an extended period of difficulty ahead.

After banking crises, house prices typically decline in real terms by 35.5% and this slump lasts on average 6 years. Now South Africa didn’t have a bank failure, so it may be that we missed the definition of “banking crisis”. However, given the pullback in credit offered along with international banking crises and property market declines, this suggests we’re in for an extended period of property market stagnation.

Steven Levitt (of Freakonomics fame) discussed an old paper of his and its usefulness in predicting US mid-term elections. This is now a 16 year-old model, which presumably could benefit with some updating for the last 16 years worth of data.

It does, currently anyway, give very similar answers to one of the biggest prediction markets operating, InTrade.com.

So, how does one not lose money with the Make a Million competition?

Don’t enter.

You are overwhelmingly like to lose money if you enter this competition. I’ve said this before, and I’ve been right before. I’m right again.

There’s also the little idea that the  structure of the Make a Million competition increases risks of  financial meltdown

Let’s look at some hard statistics to show what I mean.

Telling statistics (what they don’t show)

In the MaM presentation, the organisers include some interesting statistics about number of trades, trading activity and many other metrics.

They don’t show average returns or performance.

So let’s look at some of the numbers:

Raw return data (excluding prize money) based on 2009 MaM competition.

Suddenly the competition doesn’t look so great, does it?  (This isn’t the first time, here is my analysis of the Comedy and Tragedy that was the 2008 Make a Million competition.)

Here’s a little explanation of each of the items in the table:

Average Return

This is the return than you can expect to make on average. Yes, that’s a loss of over 10% of your investment. For all the talk about trading opportunities by the MaM organisers and sponsors, the trading result of this competition (ignoring prizes) is that more money is lost than is made.

Expected Loss

This is the total Rand amount you will lose on average (again ignoring prizes) by entering the competition. Quite a steep price. (The prize money makes the competition profitable on average, but only in a very skewed manner that only helps one person.  More on this in a bit)

Median Return

This is the return that half the entrants earned less than, and half the entrants earned more than. Half the participants lost more than 15% of their investment.

Mode Return

This is less intuitive to understand. The most common result for a entrant was to lose 9% of their starting stake.

Probability of breaking even, earning less than 10% or doubling money

Hopefully these are reasonably self-explanatory. What’s clear is that there is a high probability of doing badly, and a low probability of doing well.

Probability of winning

Ultimately your probability of winning serious money is still very low.

Great returns and manageable risk?

The MaM roadshow presentation concludes that there exist opportunities for great returns and manageable risk. I’m not sure what definition they’re using for “great returns” or “manageable risk” but in my book the returns are low and the risk is high. Look at the figures in the table above and explain how that can be interpreted any differently.

Is there any good news at all?

In fairness, there is a million rand prize available for the winner. This doesn’t change the probability of winning, the probability of earning 10%, the probability of doubling your money, the probability of losing any particular amount, but does add extra winnings to the best performer, which increases the average return considerably to 8.5% over the period. So yes, if you win, you will win. Remember the 0.2% probability of winning. In odds, that is about 500:1 against.

Winning strategies

There are some strategies than can help you win Make a Million.  They’re not really legal or ethical but you might want to know the sort of thing your competitors may be up to.

Their housing price index is more robust (and more independent) than those of the banks based off their own data and target markets. Rather than consider only the average price of houses sold in that particular month (which is a function of house price growth / decline but also how the type, condition, size and location of the houses sold that month differ from the prior month and year) they consider repeat sales where the same property has been bought and sold more than once.

This data is combined or “chain-linked” to provide a continuous measure of house price inflation over time.

House Price Inflation 2010 source: lightstone.co.za

The result of all of this data, best-in-class methodology and analysis? When Lightstone says “opportunities abound in local market” I actually listen. Since their business model is to sell information, I’m more likely to trust what they say.

It’s almost all unknown

I don’t disagree.  It’s just that almost every meaningful decision ever made is made without all the information.

Unknowns can be categorised a hundred different ways. One way is to think about:

1. Unknown past information
2. Uncertainty around the current situation or position
3. Unknown future outcomes

Even a game like chess, where the past history of the game is easily known by good players, the current position is clearly visible and all the possible moves are knowable, it is not possible have all the information about how your opponent will react to your move.

How to deal with decision making under uncertainty – part 1

Tyler suggests that gut-based decision making can be effective much of the time – and it can. It there genuinely is no time for anything more than an instinctive reaction, you probably are best going with your gut.

Even if you have plenty of time, listening to your guy to formulate an idea is a great idea. Insight comes partly from experience and the reinforced neural pathways of our learning brain. If you stop with the gut though, you are missing out. There is a tremendous amount of research showing how ridiculously badly our instincts perform in many areas, particularly those relating to uncertainty and complexity!

The way to use your instincts is to create hypotheses or theories to test and evaluate with real information and analysis. Prove that your instinct is right and you’re good to go. Try really hard and not disprove it, without coming up with better ideas, well maybe it’s worth giving it a shot anyway. Realising that your immediate feelings were deeply flawed? Priceless.

How to deal with decision making under uncertainty – part 2

Tyler also gives four suggestions if you don’t want to merely follow your instincts blindly.

1. Stop and Think
2. Ask for Advice
3. Give it Time
4. Relax and Have Fun

These probably aren’t the four worst ways to make decisions, but they’re certainly not the best.

Stop and Think is merely an invitation not to follow your gut. No news here.

Ask for Advice is actually pretty good. It’s just not the correct first step though (more on that in a moment).

Give it Time can also work, but on its own its just a different way of using your instincts. You’ll never know if, how or why you changed your mind. You also put yourself at extreme risk of confirmation bias.

Relax and Have Fun certainly sounds like a good idea, although whether it will give rise to good decisions depends very much on the time of industry and type of problem. In the same way that brainstorming is a good method for marketing and social appeal of ideas (it was invented by Alex Oxborn, an advertising manager) but not great for most other problems, relaxing and having fun offers no real promise of sophisticated decision making with uncertainty and complexity.

How should you make decisions with limited information?

I’m not going to say there is one correct way, but the following points are helpful:

• Work out, very specifically, what problem you’re trying to solve or what question you’re trying to answer. Sometimes framing the question clearly can provide its own insights. Further, if you are to work collaboratively with others, or seek advice, you’ll have to have a very clear idea of what you’re trying to achieve. Communication is critical to teamwork.
• Work out what information you have, what you need, what you could get, where you can get it and what the costs might be of obtaining this information. Now you can ask for advice because you know what you’re asking for.
• Use your instincts and initial thoughts and feelings to generate some avenues of investigation or hypotheses to test
• Actively work to generate new ideas and possible solutions or lines of investigation. Here, brainstorming might be useful as one method. Consider similar problems, examples or analogies from other people, other organisations, other industries, other countries and cultures, other time periods. This is where you get to be creative. Don’t worry too much about whether or not the ideas are workable – that comes later.
• Categorise and prioritise the information, questions and hypotheses. As you finish generating ideas, you need to start synthesizing the ideas into useful units.
• Critical evaluate the ideas, using logic, analysis, modelling and frameworks. Evaluation here requires both sides of the coin. What are the problems, the flaws, the risks, the reasons it can’t work. Then, what can we change, tweak, fix or improve to get around these problems or otherwise improve the scenario.

Throughout, ensure you keep the big picture in mind, that you remember what you’re trying to achieve. The dangers of under-analysis are only trumped by too much, misdirected meandering analysis.  Keep it focussed.

The steps above don’t necessarily have to be done in order, and typically each step can be revisited several times during the process. These ideas borrow heavily from Edward de Bono’s Six Hats of Thinking.

The point here is that complexity and uncertainty requires a more, not less, structured approach to use as much information as possible and to avoid many of the heuristic mistakes humans make. This is true in general, but even more so if you need to work in a team or need to convince external parties of your process to lend credibility to your decisions.

The headline inflation rate in July 2010 (i.e. the Consumer Price Index for all urban areas in July 2010 compared with that at July 2009) was 3,7%

The official inflation rate (i.e. the percentage change in the CPI for all urban areas in July 2010 compared with that in July 2009) was 3,7% at July 2010. This rate was 0,5 of a percentage point lower than the corresponding annual rate of 4,2% in June 2010 (i.e. the Consumer Price Index for all urban areas in June 2010 compared with that in June 2009).

From June 2010 to July 2010 the Consumer Price Index for all urban increased by 0,6%

CPI Headline July 2010 = 3,7%

So this is close to the bottom of our 3% to 6% inflation targeting range. Economic growth is struggling, unemployment is high, but we haven’t reduced interest rates? Something here is a little odd.

I’ll put another $100 in Kiva, to be “microlent” to businesses and people across the world, if the next monetary policy committee meeting doesn’t cut interest rates.

Red cars, supposedly, attract more than their fair share of traffic fines.

Turns out this is incorrect.  Snopes.com has (as usual) an excellent article on red cars, including references to research showing red cars are not more likely to be fined than other vehicles. Unfortunately, the underlying research isn’t available online (as far as I could find).

The hype and hoopla around this topic is fueled by our feelings as consumers of being charged too much big big monopoly companies. So I should start by saying that I’m not saying that we are paying too much. I’m not saying that because I don’t know enough about the costs of providing cellular services in South Africa. Maybe we are, maybe we’re not. Also, I’m not saying there aren’t monopolistic practices in the market – again I simply don’t know. Given the other stories torn from inside companies by the sharp teeth and salivating jaws of the Competition Commission, it’s understandable that many suspect consumer-unfriendly play by most large South African companies, particularly those in industries with a small number of players.

What I am saying is that most of what you read in the news about interconnect is horribly misguided.

The biggest misconception is that interconnect fees are an expense for cellular providers, and that the removal of this expense would allow them to reduce tariffs to consumers. Well, it is an expense, but it is also a source of revenue. Every time one company pays an interconnect fee, another company is receiving it.

Interconnect does not change the total amount of profit within the cellular industry. It may redistribute it a little, and there may be negative medium term competitive implications arising from interconnect, but lower interconnect won’t automatically increase profits that could allow competitive price lowering for the benefit of consumers.

TechCentral has an interesting article: Bain warns consumers not to expect cellular price cuts.  Of course, it also include some done-to-death flawed statements (whether from Bain or inserted by the zealous staff writer) such as:

Because new players have few customers at first, most calls on their networks will be to networks of other operators. High interconnection fees make it difficult for them to enter the market.

It’s not that this statement is incorrect (it is in fact correct) it’s just that it is horribly misleading because it only presents one side of the story. I’ve reworded it to provide the stunning insight:

Because new players have few customers at first, most calls to their networks will be from networks of other operators. High interconnection fees make it profitable for them to enter the market.

If you are a small cellular operator, most people calling your customers won’t also be your customers. You get to charge them an interconnect fee for most calls. You can model this in a spreadsheet (I’ve done it) and provided two basic assumptions hold, interconnect is irrelevant as a primary force. Fees in and expenses out equate .

1. “Cellphone users must make calls, on average, equally to all other subscribers independent of network.” If Cell C customers are more likely to call Cell C customers rather than a random cellphone user in South Africa, the numbers start to change. Although I don’t have info to back this assumption up, it feels reasonably robust.
2. “Customers on all networks must, on average, make the same number of calls.” This is actually where the problems arise and the true cost of interconnect exists.

Why is assumption #2 a problem? Think about the goal of competition for consumers: “Profit maximising companies see to increase volumes by lowering prices, gaining market share and thus making more profit. Provided Marginal Revenue is above Marginal Cost, companies should cut prices.”

So, what happens with interconnect fees above “true” cost of completing the call? When a company seeks to lower its prices, below that of the competition, its customers will make more calls than average. (This is intuitive and also expected from a downwards sloping demand curve.)

Company A reduces its call rates. Company A’s subscribers will make more calls (incurring interconnect expenses for Company A paying to Companies B, C and D) but customers of Company B (and C and D etc.) won’t be making more calls into Company A. Thus, Company A pays more interconnect and receives no more interconnect. Its costs have just gone up, pushing up Marginal Cost to a point where it doesn’t make sense to lower prices.

Voila – a perfect pricing system to force prices higher and higher. If Company B raises it’s prices, its subscribers will receive more calls than they make, resulting in more interconnect revenue than expenses for Company B. If the interconnect fee is sufficiently above the true cost, the reduction in profit form lower call volumes will be more than offset by the much  higher profit from interconnect fees being greater than interconnect expenses.

So interconnect fees need to come down to true cost plus a fair profit margin. It has little to do with interconnect being an expense factored into retail tariffs, but rather a function of the competitive pricing actions it encourages.