Some of the magic behind optimising Google’s search algorithms

…and a lesson about business process optimisation that you should learn

The New York Times recently posted an incredibly interesting article about Google, providing more detail about their current search algorithms and processes. Google Keeps Tweaking Its Search Engine is definitely worth a read.

Reading the article made we wonder for a moment whether Google is laying the groundwork for an ever increasingly difficult task. Increasingly difficult to the point of insurmountable in even a relatively short amount of time with their current approach. I don’t know enough about what Mr Singhal and his team does, and it’s pretty clear that are doing an incredible job and have only scraped the surface of an iceberg in terms of telling the general public (you and me!) what they get up. So I don’t know the details of what they do, but here is an important idea to generate some thought for their business, and as an example for yours.
You start with a rule, a simple rule, a good rule that ranks pages. It does a good job and is intuitive to understand. Mortals (albeit really, really smart mortals) can think through the implications of tweaking a few parameters, the weights in the calculations. Any small change can be managed with ease because the system is simple.

Then you realise that one rule isn’t enough. It’s too crude, too blunt and all that’s needed to fix some major short-comings is to introduce a second rule. Now the system is more complicated. Two, interacting rules must be considered with every change in parameter. It’s still possible to understand how changes will affect the rankings of all the millions and millions of pages ranked, but it is more difficult.

As each successful adjustment and rule-change and rule addition and new algorithm and special case and “French Revolution” and “Apple versus apple” is embedded, the system becomes more complex, less intuitive. And the number of people who can understand and appreciate it dwindles rapidly. Every change requires more thought, more consideration. The risks are greater – the system is already so finely balanced, that it is that much easier to tilt it off balance. At some point, maybe the marginal cost of making any change comes too close to the marginal benefit from the change that changes are not possible, and the State of Search Stagnates.

Some of the other avenues they seem to be exploring, based on analysing information from individual users web-habits from things like Gmail or the google-toolbar may be less susceptible. This is a new source of information, rather than just increasingly complex tweakings and additions to a system. However, the insights obtained from these new data sources must still be integrated into the search results for you and me when we search “technical business advantage” and hope to find something useful.

From the article:

‚ÄúPeople still think that Google is the gold standard of search,‚Ä? Mr. Battelle says. ‚ÄúTheir secret sauce is how these guys are doing it all in aggregate. There are 1,000 little tunings they do.‚Ä?

The lesson for the rest of us

Optimising any business process should not primarily be about making it work for now at all costs. Simple rules that have longevity and can continue to function usefully in a variety of different scenarios, over long periods of time are important. Quite frankly, a single business process of most businesses is just that, a process important to the efficient function of the business, but not the entire be-all and end-all of the business. Search is a highly competitive field, and every last ounce of performance that can be squeezed out of the algorithms is important. For search, maybe the benefits do outweigh the negatives, but for most other business decisions, it is important to optimise and structure and analyse with a long-term, sustainable, maintainable, manageable and transferable set of rules and algorithms.