A MODEL ISN’T REALITY

Any simplified model I can imagine will also therefore not match reality exactly. The closer the model gets to the real world in more scenarios, the better it is.

Not all model parameters are created equal

Part of the approach to getting a model to match reality as closely as possible is calibration. Models will typically have a range of parameters. Some will be well-established and can be set confidently without much debate. Others will have a range of reasonable or possible values based on empirical research or theory. Yet others will be relatively arbitrary or unobservable.

We don’t have to guess these values, even for the unobservable parameters. Through the process of calibration, the outputs of our model can be matched as closely as possible to actual historical values by changing the input parameters. The more certain we are of the parameters a priori the less we vary the parameters to calibrate the model. The parameters with most uncertainty are free to move as much as possible to fit the desired outputs.

During this process, the more structure or relationships that can be specified the better. The danger is that with relatively few data points (typically) and relatively many parameters (again typically) there will be multiple parameter sets that fit the data with possibly only very limited difference in “goodness of fit” for the results. The more information we add to the calibration process (additional raw data, more narrowly constrained parameters based on other research, tighter relationships between parameters) the more likely we are to derive a useful, sensible model that not only fits out calibration data well but also will be useful for predictions of the future or different decisions.

How not to calibrate a model

Scientific American has a naive article outlining “why economic models are always wrong”. I have two major problems with the story:

1. All models are wrong. Some are useful (George Box). “wrongness” isn’t a problem with a model, but lack of usefulness is. The headline demonstrates a starting point poorly informed about the point of economic models.
2. The calibration approach criticised in the article is an extremely poor way to calibrate a model. No serious researcher thinks that is the right way to calibrate a model. So the article merely creates a straw man and then demonstrates how easy it is to knock the argument over.

Calibration and back-testing on separate data sets

The right way to calibrate a model is to separate the data-set into at least two independent subsets. Firstly, the “training set” or portion from which we will calibrate our parameters to get them to match as closely as possible the data. Again, this should make use of all information available and may give rise to several competing models that appear to fit the data similarly well.

The next step is crucial. We back-test the derived models against the second subset of data. This data comes from the same reality (perhaps a different time period) as used to calibrate the model, but the model won’t trivially match the data because none of that data was used to calibrate the model in the first place.

The importance of back-testing

Back-testing is critically important in the model building process, but back-testing against the same data used to calibrate the model is worth than useless (since it takes time and effort and an create a false sense of accuracy or reliability in the model.) Separating the data into two or more subsets is absolutely required, although it has the unfortunate side-effect of reducing the size of the data-set available for calibration.

Yes, it really matters.

A common example of the dangers of bad models fitting data well is with Economic Scenario Generators. These simulate economic scenarios to be used in valuing complex financial securities. If a model is properly calibrated, it will recreate the observable market prices of a wide range of instruments. However, the model could be a black-box neural network, a carefully constructed theoretical model with plausible relationships and constraints, or the proverbial ten thousand (possibly inebriated) monkeys. If all three models are perfectly calibrated to observable market prices, is any of the models inferior to any of the others?

Clearly the answer is yes, but only when it comes to extrapolation. I have far more confidence in the model’s ability to create “market consistent” valuations for instruments that do not have observable prices in the market if I understand how the mechanics of the model make sense on a level other than pure calibration.

Trivial 3 point example

Perfect fit of two models to 3 data points

The example above shows a perfect of a quadratic and cubic model to 3 data points. From this graph, both models appear exactly the same.

However, if we use the model to extrapolate to future time periods, the results are very different. Without additional data to back-test the results on, it’s not possible to tell whether either or any of these models is appropriate, but clearly both can’t be correct.

Extrapolation of the two models shows divergent results

But I know it doesn’t make me efficient. Switching between tasks takes time. You forget important details, and struggle to get the depth of understanding and focus required for complex issues. The cross-pollination of ideas and solutions doesn’t come close to making up for these drawbacks.

From the research:

Descriptive evidence suggests that judges who keep fewer trials active and wait to close the open ones before starting new ones, dispose more rapidly of a larger number of cases per unit of time. In this way, their backlog remains low even though they receive the same workload as other judges who juggle more trials at any given time.

Did you read those magic words? “…backlog remains low…” I don’t know anyone who doesn’t wish for the luxury of a shorter backlog of work.

The paper itself is fairly complex, analysing theoretical models of human task scheduling.  You should probably add it to your pile of things to read in the middle of other work.

Their housing price index is more robust (and more independent) than those of the banks based off their own data and target markets. Rather than consider only the average price of houses sold in that particular month (which is a function of house price growth / decline but also how the type, condition, size and location of the houses sold that month differ from the prior month and year) they consider repeat sales where the same property has been bought and sold more than once.

This data is combined or “chain-linked” to provide a continuous measure of house price inflation over time.

House Price Inflation 2010 source: lightstone.co.za

The result of all of this data, best-in-class methodology and analysis? When Lightstone says “opportunities abound in local market” I actually listen. Since their business model is to sell information, I’m more likely to trust what they say.

It’s almost all unknown

I don’t disagree.  It’s just that almost every meaningful decision ever made is made without all the information.

Unknowns can be categorised a hundred different ways. One way is to think about:

1. Unknown past information
2. Uncertainty around the current situation or position
3. Unknown future outcomes

Even a game like chess, where the past history of the game is easily known by good players, the current position is clearly visible and all the possible moves are knowable, it is not possible have all the information about how your opponent will react to your move.

How to deal with decision making under uncertainty – part 1

Tyler suggests that gut-based decision making can be effective much of the time – and it can. It there genuinely is no time for anything more than an instinctive reaction, you probably are best going with your gut.

Even if you have plenty of time, listening to your guy to formulate an idea is a great idea. Insight comes partly from experience and the reinforced neural pathways of our learning brain. If you stop with the gut though, you are missing out. There is a tremendous amount of research showing how ridiculously badly our instincts perform in many areas, particularly those relating to uncertainty and complexity!

The way to use your instincts is to create hypotheses or theories to test and evaluate with real information and analysis. Prove that your instinct is right and you’re good to go. Try really hard and not disprove it, without coming up with better ideas, well maybe it’s worth giving it a shot anyway. Realising that your immediate feelings were deeply flawed? Priceless.

How to deal with decision making under uncertainty – part 2

Tyler also gives four suggestions if you don’t want to merely follow your instincts blindly.

1. Stop and Think
2. Ask for Advice
3. Give it Time
4. Relax and Have Fun

These probably aren’t the four worst ways to make decisions, but they’re certainly not the best.

Stop and Think is merely an invitation not to follow your gut. No news here.

Ask for Advice is actually pretty good. It’s just not the correct first step though (more on that in a moment).

Give it Time can also work, but on its own its just a different way of using your instincts. You’ll never know if, how or why you changed your mind. You also put yourself at extreme risk of confirmation bias.

Relax and Have Fun certainly sounds like a good idea, although whether it will give rise to good decisions depends very much on the time of industry and type of problem. In the same way that brainstorming is a good method for marketing and social appeal of ideas (it was invented by Alex Oxborn, an advertising manager) but not great for most other problems, relaxing and having fun offers no real promise of sophisticated decision making with uncertainty and complexity.

How should you make decisions with limited information?

I’m not going to say there is one correct way, but the following points are helpful:

• Work out, very specifically, what problem you’re trying to solve or what question you’re trying to answer. Sometimes framing the question clearly can provide its own insights. Further, if you are to work collaboratively with others, or seek advice, you’ll have to have a very clear idea of what you’re trying to achieve. Communication is critical to teamwork.
• Work out what information you have, what you need, what you could get, where you can get it and what the costs might be of obtaining this information. Now you can ask for advice because you know what you’re asking for.
• Use your instincts and initial thoughts and feelings to generate some avenues of investigation or hypotheses to test
• Actively work to generate new ideas and possible solutions or lines of investigation. Here, brainstorming might be useful as one method. Consider similar problems, examples or analogies from other people, other organisations, other industries, other countries and cultures, other time periods. This is where you get to be creative. Don’t worry too much about whether or not the ideas are workable – that comes later.
• Categorise and prioritise the information, questions and hypotheses. As you finish generating ideas, you need to start synthesizing the ideas into useful units.
• Critical evaluate the ideas, using logic, analysis, modelling and frameworks. Evaluation here requires both sides of the coin. What are the problems, the flaws, the risks, the reasons it can’t work. Then, what can we change, tweak, fix or improve to get around these problems or otherwise improve the scenario.

Throughout, ensure you keep the big picture in mind, that you remember what you’re trying to achieve. The dangers of under-analysis are only trumped by too much, misdirected meandering analysis.  Keep it focussed.

The steps above don’t necessarily have to be done in order, and typically each step can be revisited several times during the process. These ideas borrow heavily from Edward de Bono’s Six Hats of Thinking.

The point here is that complexity and uncertainty requires a more, not less, structured approach to use as much information as possible and to avoid many of the heuristic mistakes humans make. This is true in general, but even more so if you need to work in a team or need to convince external parties of your process to lend credibility to your decisions.

1 The Competition Commission is necessary

Monopolies serve to increase prices for consumers. In Monopoly, the “rents” charged are instantly higher as soon as a player has a monopoly on property in a certain area.

Worse than the increase in prices and decrease in supply, the additional profit for suppliers is not equal to the cost to consumers from higher prices, resulting in an overall “dead weight loss of monopoly” or an overall cost to society.

To date, the Competition Commission has focussed on cartels and price collusion. We can only hope soon they will consider issues of local loop unbundling to provide real competition to the effective monopoly Telkom has on ADSL lines.

2 The easy availability of credit is a useful and dangerous tool

An important consideration in Monopoly is to prevent competitors from owning property (so that you don’t have to pay rent on as many properties as possible) and definitely don’t get a monopoly (so they can increase rents and improve the properties, increasing rents even further).

A key way of doing this is by buying as many properties as possible, even if you mortgage many of them with a fresh injection of cash from the bank and very reasonable repayment terms. Leverage, or borrowing, can provide cost effective (especially after tax considerations) financing for profitable projects. It can be cheaper to raise than equity and increases returns to shareholders.

Leverage also creates significant risks, both in Monopoly and real life. Overextending can make you more likely to end up bankrupt. Also, by artificially increase the supply of funds for purchasing property while keeping the supply constant, the price of property will increase dramatically with distorting impacts on the allocation of resources within an economy.

3 Trading is good for those who trade

Every time two people (or organisations, or countries) trade, both parties are better off than they were before. (A hint, otherwise they wouldn’t have traded.) The trading parties are each better off, and the non-trading competitors are worse off, because a potential trade is now off the table and they are not in a better position otherwise.

There are some plausible arguments for trade restrictions in very particular circumstances. For example, I think I buy the idea of nurturing infant industries, but only if there is a reasonable chance that the industries won’t only grow as far as pimply, sulky adolescents with no drive or ambition.

Other than that, trade is good as it increases prosperity on average, which is one important way to lift more families incomes above the poverty line.

4 Liquidity risk can be the end of you

While buying as much property as possible and mortgaging some to provide additional funds will generally work in your favour and allow you greater control over the board, it also introduces liquidity risk. If you don’t have the funds to pay rent or taxes or fines when they unexpectedly fall due, you will be forced into a fire-sale of houses and hotels at way below replacement cost. You may need to mortgage even more properties, reducing your income generating ability and possibly sending you into a debt trap where your income isn’t sufficient to meet your expenses.

Always watch your liquidity. It has almost become a truism that more businesses fail because of liquidity than solvency or profitability.

5 The world is full of randomness, but it always helps to calculate the numbers

In Monopoly, the player “roles the dice and moves the mice” so to speak. There is plenty of luck around and skill can seem to disappear in an avalanche of random numbers.

Nassim Taleb (of Fooled By Randomness fame) would suggest that in everyday life we underestimate risk and overestimate skill. I tend to agree with this overall comment. It doesn’t mean that we should try to use our skill as much as possible.

Did you know that certain properties are relatively more valuable than others? The properties just after Jail are more likely to be hit than others, since several cards and one square on the board send the player to jail. Similarly, several chance cards require the player to advance a certain way, making the squares immediately after a chance square less likely to be hit. The cost to benefit of houses on certain squares are better than others, and the best bang for buck is somewhere between 2 and 3 houses per property.

It’s important to be aware that risk and uncertainty exists, but it’s more important to understand risk in a way that allows informed, risk-aware decisions to be made.

Identify, measure, mitigate and monitor risk!

Are all the lessons good?

Slightly tongue in cheek, there are a few other lessons of dubious validity:

1. Stay in jail and make more money
2. You get to keep the money if the bank makes an error in your favour
3. You can take as long as you want to repay your mortgage and only have to pay 10% interest
4. If the bank runs out of money, it can always photocopy some more
5. When the game is over, you can forget about the debt you have and walk away

It’s that nobody else wants it. The resale value is much lower than other vehicles, and the risks and costs of stealing it are no lower.

Unlike the conclusion around high risk vehicles in my post on hijacking, this actually means you should be safer in a pink car. Just not safer from ridicule.

Dutch professor, Ben Vollaard, studied theft rates for vehicles as part of his research area of the economics of crimes. The data covered 109 vehicles from 2004 to 2008. Not the largest sample size, but enough to start thinking.

From the research:

a car thief interested in making money can be expected to go for cars with the highest resale value, i.e. cars in the most popular exterior colours. Colour is key in the car market. A car in silver or yellow goes for the same price at the dealership, but the resale value greatly differs between the two. The National Auto Auction Association estimates that on average a used car in a popular colour sells for $1,000 more than the same car with a less desirable colour (NAAA 2010).

What colours people want

DuPont provides a list of the global popularity of vehicle colours:

Top Ten Global Vehicle Colors

1. Silver – 25 percent
2. Black – 23 percent
3. White – 16 percent
4. Gray – 13 percent
5. Blue – 9 percent
6. Red – 8 percent
7. Brown/Beige – 4 percent
8. Green – 1 percent
9. Yellow/Gold – 1 percent
10. Others – <1 percent

Pink, along with purple and polka dots and everything else represents less than 1% of the vehicles on the road.

This is a global study, but it ties with my sense of colour popularity in South Africa.  Interestingly, in India, red is in third place with 16.1%. Green is the second most popular colour in Russia with 18.2% of all cars.

What criminals want

The link between market popularity and theft risk is obvious:

Theft risk by colour (%), cars up to three years old, 2004-2008, the Netherlands. Source Ben Vollaard

The risk of theft for black, silver and blue cars is notably higher than for other, less popular colours. Not a single pink car was stolen in the sample.

The research also shows that there is more at work here than the visibility of the cars. Red cars have decreased in popularity over time. As the popularity has decreased, so has the risk of theft. The cars haven’t become less visible to police over the period, but as their popularity changes, so the resale value declines. This reduced value to thieves makes them less attractive and therefore less risky for the owner (not the criminal…).

Should you want what others don’t?

If you plan to keep the car forever, and if you can stomach driving a pink vehicle, maybe you should consider pink to be the new black. If you are planning to ever have to sell the vehicle, you’ll be in the same (pink) boat as the criminals.

It seems that NERSA has been looking at this and has discovered some areas where Eskom was incorrectly estimating their future costs. The numbers are large, but not so large in the greater budget of Eskom. This is more like a snowflake on an iceberg than anything meaningful, but it does show the value of looking at the real issues.

The hype and hoopla around this topic is fueled by our feelings as consumers of being charged too much big big monopoly companies. So I should start by saying that I’m not saying that we are paying too much. I’m not saying that because I don’t know enough about the costs of providing cellular services in South Africa. Maybe we are, maybe we’re not. Also, I’m not saying there aren’t monopolistic practices in the market – again I simply don’t know. Given the other stories torn from inside companies by the sharp teeth and salivating jaws of the Competition Commission, it’s understandable that many suspect consumer-unfriendly play by most large South African companies, particularly those in industries with a small number of players.

What I am saying is that most of what you read in the news about interconnect is horribly misguided.

The biggest misconception is that interconnect fees are an expense for cellular providers, and that the removal of this expense would allow them to reduce tariffs to consumers. Well, it is an expense, but it is also a source of revenue. Every time one company pays an interconnect fee, another company is receiving it.

Interconnect does not change the total amount of profit within the cellular industry. It may redistribute it a little, and there may be negative medium term competitive implications arising from interconnect, but lower interconnect won’t automatically increase profits that could allow competitive price lowering for the benefit of consumers.

TechCentral has an interesting article: Bain warns consumers not to expect cellular price cuts.  Of course, it also include some done-to-death flawed statements (whether from Bain or inserted by the zealous staff writer) such as:

Because new players have few customers at first, most calls on their networks will be to networks of other operators. High interconnection fees make it difficult for them to enter the market.

It’s not that this statement is incorrect (it is in fact correct) it’s just that it is horribly misleading because it only presents one side of the story. I’ve reworded it to provide the stunning insight:

Because new players have few customers at first, most calls to their networks will be from networks of other operators. High interconnection fees make it profitable for them to enter the market.

If you are a small cellular operator, most people calling your customers won’t also be your customers. You get to charge them an interconnect fee for most calls. You can model this in a spreadsheet (I’ve done it) and provided two basic assumptions hold, interconnect is irrelevant as a primary force. Fees in and expenses out equate .

1. “Cellphone users must make calls, on average, equally to all other subscribers independent of network.” If Cell C customers are more likely to call Cell C customers rather than a random cellphone user in South Africa, the numbers start to change. Although I don’t have info to back this assumption up, it feels reasonably robust.
2. “Customers on all networks must, on average, make the same number of calls.” This is actually where the problems arise and the true cost of interconnect exists.

Why is assumption #2 a problem? Think about the goal of competition for consumers: “Profit maximising companies see to increase volumes by lowering prices, gaining market share and thus making more profit. Provided Marginal Revenue is above Marginal Cost, companies should cut prices.”

So, what happens with interconnect fees above “true” cost of completing the call? When a company seeks to lower its prices, below that of the competition, its customers will make more calls than average. (This is intuitive and also expected from a downwards sloping demand curve.)

Company A reduces its call rates. Company A’s subscribers will make more calls (incurring interconnect expenses for Company A paying to Companies B, C and D) but customers of Company B (and C and D etc.) won’t be making more calls into Company A. Thus, Company A pays more interconnect and receives no more interconnect. Its costs have just gone up, pushing up Marginal Cost to a point where it doesn’t make sense to lower prices.

Voila – a perfect pricing system to force prices higher and higher. If Company B raises it’s prices, its subscribers will receive more calls than they make, resulting in more interconnect revenue than expenses for Company B. If the interconnect fee is sufficiently above the true cost, the reduction in profit form lower call volumes will be more than offset by the much  higher profit from interconnect fees being greater than interconnect expenses.

So interconnect fees need to come down to true cost plus a fair profit margin. It has little to do with interconnect being an expense factored into retail tariffs, but rather a function of the competitive pricing actions it encourages.

Commonly known tweaks include turning off screen updating (see the code below for an example of how this works) and then updating the statusbar to show the user progress (also in the code below).

However, what often frustrates me is the inelegant ways in which a user can exit a macro which could take hours to run. I struggled a little to find the approach demonstrated below so I thought it might be useful for others.

Basic premise is to use error handling to tell Excel to be on the lookout for keyboard cancellations (escape or ctrl-break) and then handle these gracefully.

Private Sub RunLongMacro()

'declare count variables

Dim count As Integer

Dim Total As Integer

Total = 10

'Set the action on an error. In this case go to the label "handleCancel"

On Error GoTo handleCancel

'Turn off screenupdating to increase run speed and start statusbar reporting to user

Application.ScreenUpdating = False
Application.Calculation = xlCalculationManual
Application.StatusBar = "Initiliasing..."

'Tell VBA what to do when a cancel key is pressed (Escape or CTRL-BREAK)

'In this case handle the error (Rather than ignore it or present standard error)

Application.EnableCancelKey = xlErrorHandler

'Start the main calculation loop

For count = 1 To Total

'Some additional code might be included here

'Then we calculate the model results

Application.Calculate

'and update the status bar with the count

Application.StatusBar = count & " / " & Total

Next count

'When we're done, we reset the statusbar and screenupdating

Application.StatusBar = False

Application.ScreenUpdating = True

'Next part is the code to handle errors, starting with the specified label

handleCancel:

'If there isn't an error, just skip to the end

'This ensures a single exit point

If Err.Number <> 0 Then

If Err.Number = 18 Then

If MsgBox("You pressed Escape or CTRL-BREAK. Do you want to quit?", vbYesNo, "Quit?") = vbNo Then

Resume

End If

Else: MsgBox "Oops. We encountered an error"

End If

End If

Application.StatusBar = False
Application.ScreenUpdating = True
Application.Calculation = xlCalculationAutomatic

End Sub

A common response though was that Lebanon isn’t yet ready for that, because rates are so low and nobody is prepared to change their approach. I accept that changing the “way things are done” in a fundamental way takes time and courage, but I expect that some players will start collecting the data, doing the analysis and improving their pricing in the next few years. By 2013, the market here will not be the same. The advantages across general insurance, banking, sales and cross-selling are simply too great. The techniques available are fantastic and can be implemented quite easily.

I’ve given the official press release below, and presentation ACAL GI Pricing 2008 (pdf version) is available under Resources on this site.

Insurance companies can generate a competitive advantage through accurate ratemaking, systematic risk-adjusted pricing, and careful analysis of policyholder price sensitivity at renewal dates. Single variable techniques can provide valuable insights into risk factors, but do not perform well in the presence of multiple drivers of risk.

Generalised Linear Modelling (GLM) is the preferred approach for robust, multivariate analysis of claim severity and frequency modelling. GLM can model several rating factors simultaneously, including interactions between different rating factors on risk. It is used extensively in the UK, the US and other highly competitive and developed insurance markets.

Judgement and experience are required when assessing different models and interpreting the diagnostic tests used to ensure accurate and robust results. A good model can make dramatic improvements in the separation of high and low risk policyholders.

These advanced approaches all have increased data requirements. Companies looking to reap the rewards of improved ratemaking will need to develop the databases and systems to store exposure, claims and rating factor data. There is a range of software available to perform the statistical analysis, from expensive purpose-built systems to freely available, open-source statistical platforms.

Successful implementation of an advanced rating system depends on commitment of key staff to the project and the inclusion of marketing, underwriting, legal, IT and actuarial skills in the project team. Market characteristics and reluctance to change are constraints to the adoption of advanced techniques. These have been faced and overcome in many other markets. It is only a matter of time before insurers must use these techniques even to maintain their competitive position.  Early movers will enjoy an improvement in their competitive position, market share and profitability.