Âé¶¹´«Ã½

Technology

Engines of the future: Google and beyond

By Michael Brooks

17 November 2010

Âé¶¹´«Ã½. Science news and long reads from expert journalists, covering developments in science, technology, health and the environment on the website and the magazine.

All eyes are on Google

(Image: Press Association Images)

Read more: Engines of the future: What’s next in internet search?

The web’s top search engine has got rich from cleverly picking the right few pages out the trillions online. What kind of upstart could usurp its crown?

Google is synonymous with web search. Some 60 per cent of surfers use it, performing roughly . All these clicks combined to earn the company a cool $6.5 billion profit last year. But if you dream of muscling in on some of this fortune you should be aware of what you are letting yourself in for.

Building a search engine is like compiling a searchable index for a . The index has to be contained on a limited number of servers, and each of its entries must be stored in such a way that the whole thing can be searched in milliseconds. Then you must face the fact that the web’s contents constantly change and that most searches are frustratingly vague – just three words long, on average.

We might be vague but we are also demanding: we want our first 10 results to include the information or site we are looking for. What’s more, we want these results to appear near-instantaneously and, of course, for free. Still want to take on Google?

To understand why Google is so dominant, you need to grasp the basics of web indexing. Search engines use “web crawler” software to trawl through online content, remembering not just the words but also how often they crop up. They also make use of…

Sign up to our weekly newsletter

Receive a weekly dose of discovery in your inbox. We'll also keep you up to date with Âé¶¹´«Ã½ events and special offers.

Sign up

To continue reading, today with our introductory offers

or

Existing subscribers

Sign in to your account
Piano Exit Overlay Banner Mobile Piano Exit Overlay Banner Desktop