Technology - Let a Hundred Search Engines Bloom

With its silly name and its spare, mostly white homepage, seems like an unlikely harbinger of change on the Web. But Google's search technology has earned it eminence among a new crop of high-end search engines.

The announcement in late June that Google will replace Inktomi as Yahoo's consumer search provider leads a surge in boutique search technology that's edging onto the turf of the one-size-fits-all engines.

Yahoo still dominates the list of popular search engines. But the market has increasingly opened to newcomers as the big portals - most of which started out as search engines - outsource searching functions. uses LookSmart and Inktomi, America Online uses Inktomi, Lycos uses Fast Search & Transfer, and Netscape Communications uses Google, according to

At the same time, surfers fed up with irrelevant or outdated search results are turning to new search engines that differentiate themselves from the established ones. Some examples:

· TopClick offers private searches without cookies and unsolicited banner ads, and it doesn't sell customer information.

· ranks search results based on Web site ratings.

· is a new metacrawler that grabs results from 24 search engines.

· Copernic offers downloadable search software in multiple languages.

· lets users search Web documents written in wireless mark-up language.

· DolphinSearch is a corporate-search product based on technology modelled after pattern recognition used by dolphins.

Some new companies also offer alternatives to traditional search engines with big promises that, so far, haven't materialised. For example, WebTop (formerly Dialog) claims to be launching an "Internet searching revolution" with its WebCheck desktop search tool, which lets people highlight sentences or paragraphs in e-mail messages, Microsoft Word documents or on the Web to find relevant information online. Similar capabilities are available with and Autonomy, which promises to "render traditional search engines obsolete" with its Kenjin technology.

With this stream of alternative search engines surfers can now tailor their searches. They could use Yahoo, for instance, for general searches and to see which sites other surfers have found useful.

For Google, the Yahoo announcement is a major profile boost. "This Yahoo deal is a significant contribution to revenues," says Google President Sergey Brin. "And more important, a validation of our technology."

Google's privately held status may have provided leverage to offer Yahoo a better deal than the more established publicly traded Inktomi, analysts speculate. "It's hard to pursue Yahoo," says Richard Pierce, COO of Inktomi. "But we're a scrappy bunch. We're going to look for an opportunity to win them back."

Google resembles Yahoo in its early days. Like Yahoo, Google was founded by two Stanford University students - Brin, now 26, and CEO Larry Page, 27 - who left school to start the business. Since its founding, searches on the California.-based company site have increased 20 per cent a month, Brin says.

The company has staked its future on assembling the largest database of searchable Web pages. Late last month Google announced that its searches encompass more Web pages than any other search engine - averaging more than 1 billion. That includes 560 million full-text indexed pages and 500 million partially indexed pages. By contrast, Inktomi's database includes 500 million pages, double those of AltaVista and Excite. And several magazines have awarded "best search engine" to Google's technology.

"Overall, Google is doing very well," says Steve Lawrence, research scientist at NEC Research Institute in New Jersey. "They're indexing more of the Web than anybody else," he says, "and they rank highest on the independent surveys," including the consumer survey conducted earlier this year by research firm NPD New Media Services.

"Google has defined themselves to be one of the leading search companies out there," says Phu Hoang, VP of engineering for Yahoo's e-commerce division. "They've won numerous awards. We've had users asking us about them and we looked into it and they really have a good search relevance."

Relevance is the operative term.

In addition to prowling directories assembled by humans, most searches hunt for Web keywords without checking whether they're relevant to the search. In an attempt to measure relevancy, Google analyses sites that link to pages containing the keywords and examines how prominent the keywords are on the pages. (For example, a search on Google for "Olympic fencing" produced links to the Georgian Olympic team's home page, AOL's Total Sports Olympic preview and a CBS SportsLine history of fencing. The top result for the same search on's portal was a page from, an online magazine for the Church of Scientology.) Google's link analysis and simple user interface have been emulated by AltaVista and ExciteAtHome, among others.

The next new search star could already be on the horizon. Direct Hit superseded Inktomi as HotBot's main search provider in 1998. Direct Hit technology, which ranks sites based on how many clicks they receive, helps power search results on Ask Jeeves, iWon and Lycos, according to SearchEngineWatch.

Fast Search Transfer, an Oslo -based site, could become the search-engine flavour of next month. The company, which uses "spidering" software that continually patrols the Web for new and updated sites along with ultrafast indexing technology, has partnered with Dell Computer in an attempt to overtake Google with the world's biggest search database.

Currently encompassing 80 million Web pages, Fast Search expects to grow to 200 million by the end of the summer. In June, Lycos stopped doing its own crawling and started using Fast Search, in conjunction with Direct Hit for some searches. Fast Search now powers about 20 sites, according to Danny Sullivan, editor of SearchEngineWatch.

Northern Light, another high-end search provider, offers its patented technology to gauge Web-page relevance. It has also teamed with Medialink to offer a service that monitors thousands of news outlets including newspapers, TV networks, Web sites and collect documents or news spots on specific topics.

Northern Light "has big coverage of the Web, but they're not on the [top search engines] chart because they're not heavily used," says Sullivan. "They appeal to professional researchers. It may be that they're going to succeed as a tool for serious searchers."

The wave of the future could lie with sites such as, which customises search results. Search sites may follow the lead of the big portals and personalise searches to increase surfer loyalty.

Niche sites that search within a narrow universe of the Web, such as, which is dedicated to computer products and services, and, which deals with legal questions - are likely to become more popular.

The question is whether all the search boutiques can generate enough revenue to survive. In the meantime, frustrated Internet seekers will continue to enjoy a panoply of spiders, relevance rankers and database detectives.

"Two years from now," says Sullivan, "we may have someone who is a new Google come in and want to make a name for itself."

Published by arrangement with The Industry Standard (US)

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about AltavistaAmerica OnlineAOLAsk JeevesCBS CorporationDell ComputerFast Search & TransferGo.comGoogleInktomiiWonLookSmartLycosMicrosoftMSNNECNorthern LightStanford UniversityTotal SportsYahoo

Show Comments