Google - The Silicon Valley Monolith

DLD friend Steven Levy explores Google's anatomy and looks under the hood of the legendary search engine in an article of the March 2010 edition of Wired Magazine. Browse through the concentrated juice of his exclusive look at the web's ultimate algorithm in the subsequent recapitulation.

This year Google will introduce 550 or so improvements to its fabled algorithm. The decisions made at the weekly Search Quality Launch Meeting at the Mountain View headquarter will wind up affecting the results you get when you use Google's search engine. After a solid decade of search market dominance, Google holds a commanding market share and still is synonymous with the verb search. Still, Google isn't ready to rest on its laurels; its competitors aren't ready to concede defeat. A few companies are challenging the central premise: that a single search engine - through technological wizadry and constant refinement - can satisfy any possible query. Facebook launched an early attack with its implication that information curation by friends is preferred to an anonymous formula. Twitter's constant stream of update introduced the concept of real-time search. Yelp helps to find practical stuff by crowdsourcing the ratings. Individually no threat, together they hint at a wide-open future of search - one that isn't dominated by a single engine.

Still, the biggest threat is Microsoft's search engine Bing. Microsoft's share of the US search market is about 11 percent - a number that will be more than double once regulators approve the deal to make Bing the search provider for Yahoo. Bing has focused on unique instances where Google's algorithm doesn't always satisfy: flight schedules and fares, the health, reference, and shopping sectors.The idea is that people will get used to tapping a different search engine for some queries as they come up with niches in which Bing excels.

However, when it comes to the simple task of taking a search term and returning relevant results, Google is miles ahead. How come?

Google's algorithm was initially based on the Pagerank, Google cofounder Larry Page's brilliant insight to rate pages based on the number of importance of links that pointed to them - to use the collective intelligence of the Web itself - to determine which sites were most relevant. The exploitation of more certain signals, contextual clues improved the relevancy ever since: the Web Page title, anchor text, synonym learning and contextualization; freshness; location and much more. Some of the most important signals come from Google itself: the data people generate when they search turns out to be an invaluable resource in discovering new signals and improving the relevancy.

At any moment, dozens of these changes are going through the testing process - voluntarily by the professional testers and random users who unwittingly participate in its constant quality experiments.

Steven Levy concludes that even if there is a paradigm shift in search, Google's algorithms will absorb that, too: "That's why Google is such a fearsome competitor; it has built a machine nimble enough to absorb almost any approach that threatens it - all while returning high-quality results that its competitor's can't match."