Gathering competitive intelligence over the Internet is a deadly serious game that can be played in many ways. There's a wealth of information readily available, and it can offer a surprising amount of insight into the competition's next move. Savvy companies are keeping close tabs on competitors in a variety of ways.

Ross Stapleton-Gray is the head of competitive intelligence at Sandstorm Enterprises Inc., a Cambridge, Mass.-based computer security firm. His job is to watch companies whose products might knock Sandstorm's out of the market. He spends his days trolling the Internet for often small bits of information that might add up to deeper insights.

In the past, information in local newspapers rarely traveled farther than the town line. Now, the Internet often makes information from those newspapers widely available to anyone who wants it. "We're following a company in Texas," Stapleton-Gray explains. "The local Texas business rag follows companies in town. They cover plenty of stuff that doesn't come out in press releases. The latest was a puff piece about how this company saw itself as an acquisition target."

That tidbit helped him understand how the company's management may be planning for the next year; trying to be acquired is a different game than trying to remain independent.

But how do you go about getting competitive intelligence? It turns out that there are lots of potential sources available on the Internet. Though a given Web site may be intended for one particular purpose, it may also reveal other useful information about companies and individuals.

Often, the interesting information about a company isn't on its Web site. Stapleton-Gray, for instance, says he enjoys watching job-posting boards for news about companies like his. While the descriptions are often intentionally vague, many times, they can't completely conceal a company's identity from a skilled reader.

"I was following a company locally, and on a local mailing list, someone wrote, ‘I'm looking to join a company,'" Stapleton-Gray says. "His vague description seemed to match a company I was watching. Given that the start-up would be competition for us, I kept in contact with him over the next several months. How's that company coming? It hasn't got a prayer."

One executive who asked for anonymity says, "I watch our competitor on a regular basis and keep a running tally of who's on their staff or not." When he sees good people, he considers hiring them. If they don't match his needs, he passes their names on to a headhunter. He says that anyone "peeled off the competitors isn't a bad thing."

The marketplace is responding to the need for competitive intelligence. Web sites that focus on providing specialized data are popping up, along with products that help people search the Internet for information about rivals.

John Fisk is director of business development at Caesius Software Inc., a Seattle company that has developed a Web-searching tool called WebQL. The product is a sophisticated engine that will download a collection of Web pages and search for particular patterns. "The functions or commands are kind of a combination of Perl and some regular expressions to find patterns in the HTML," he explains.

Fisk says that some Fortune 500 companies use his tool to watch their competitors. "Let's say you had a retail site out there and you were competing with the big brick-and-mortar and online sites. We've already written queries that track the product mix, the price or the percentage discounts. We can categorize it by [stock-keeping units] or product or anything," he says.

Once these scripts are written, they can be run periodically to watch how businesses change prices and discounts. Book authors, for instance, have been known to write scripts that visit every hour and record their books' positions on the best-seller list.

Another firm, Inc. in Santa Clara, Calif., also writes proprietary scripts that download information for its customers from competitors' Web sites and then produces reports. RivalWatch focuses on adding some analysis and presenting the data in a useful form. It specializes in surveying digital commerce sites so customers can be sure they're pricing items competitively.

"We present the data in a form [that let's you] make a business decision," says Ann Hsu, vice president for business development and a co-founder of RivalWatch. "Just giving you a spreadsheet with the prices of 10,000 products is not helpful. Only 5,000 are a direct overlap with your site, and out of these, we summarize the price differences. So you can see [that one group of] products is competitive, but these [others] are 25 percent higher."

RivalWatch is expanding the types of analysis it performs. Many clothing retailers, for instance, can't compare their products directly with competitors' because their products use slightly different fabrics, colors and designs. RivalWatch offers scripts that break down information based on the product type and price range.

"We let you know, for instance, how many shirts we have between US$15 and $25," says Hsu. "This kind of analysis is traditionally done, but it's harder. No one can go into a store and count how many shirts are available."

Sucking Up the Data

Hoover's Inc. is one of the preeminent online services that offers free and subscription-only information about corporations around the country. Austin, Texas-based Hoover's bundles basic news with its own information about thousands of companies.

Subscribers, for instance, can look at a corporate profile marketed under the name "Hoover's Intelligence Monitor" that includes information about a company's managers, products and sales.

Hoover's also offers customized analysis that compares a company's statistics against those of any three rivals. For example, you could use it to see that a firm's debt is significantly higher or its net profit margins are lower than competitors'. These figures can be compiled on the spot using the information in Hoover's database.

"The advantage we have is that we have a database of more than 50,000 companies that we can sift and sort to get the exact information for a competitor," says Laura Raun, an executive producer at Hoover's. The company is counting on the quality of its database and the thoroughness of the individual company reports it prepares in-house.

A number of other companies are also aiming at building similar portals for business users seeking competitive intelligence. Many of the old standards like Dow Jones & Co. and Moody's Corp. are on the Web, while new firms such as Inc. and Inc. are trying to muscle into the market. Still, the quality and depth of the information varies widely among the sites, which often take different approaches to gathering information.

Closing Open Doors

The growing interest in intelligence-gathering has also prompted a corresponding fascination with blocking access to information to prevent competitors from spying.

Computer security continues to be a hot area for research and investment, and corporations are looking at ways to minimize and control the information getting out from behind their walls. Some corporations scan outgoing e-mail for sensitive documents, while others monitor Web site use. Still, loopholes can emerge where you least expect them.

Simson Garfinkel, author of the book Database Nation: The Death of Privacy in the 21st Century (O'Reilly & Associates, 2001), advocates keeping a careful watch on Web sites, file transfer protocol (FTP) servers and the other ways companies share information. Often, people don't realize how much is leaving through forgotten doors, he says.

"I think that companies are exposing far more proprietary information than they ever imagined," Garfinkel says. "Internet search engines like Google make it easy to find it faster than ever. If you search for ‘secret' or ‘proprietary,' you'll find real secret or proprietary documents."

Getting such documents back can be impossible once they leave a company. Google, for instance, keeps a cache of all pages it visits. Deleting a file from a Web site or an FTP server won't remove it from Google's cache. Web sites are often visited by hundreds of Web-crawling indexing robots, and it's impossible to know which ones have saved copies of documents.

Of course, businesses also have to balance the need for secrecy with the need to communicate. Locking up all of the information on Web sites and FTP servers may seem like good practice until it starts to make life difficult for legitimate customers.

For instance, you can make life simpler for spying rivals by making prices easily available on the Web, but at the same time, you're encouraging browsing by potential customers. After all, forcing people to contact a sales force to get a price may drive away more business than it protects.

Hsu says businesses "have all these constituents they're trying to serve and only one party they want to protect against - that's their competitors. But there are so many other parties that you want to give [information] to, so you can't really hold it back. In business, you often choose to gain customers rather than keep away rivals."

Wayner, a freelance writer in Baltimore, is the author of Free for All, a book about the free software movement (Harperbusiness, 2000). Contact him at

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about Amazon.comDow JonesGoogleHoover'sO'Reilly & AssociatesRAGReillySandstorm Enterprises

Show Comments