E-Commerce (A Special Report): The Rules --- Law: Are Bots ...



E-Commerce (A Special Report): The Rules --- Law: Are Bots Legal? --- Comparison-shopping sites say they make the Web manageable; Critics say they trespass

By Phyllis Plitch

09/16/2002

The Wall Street Journal

R13

(Copyright (c) 2002, Dow Jones & Company, Inc.)

FOR A FEE, Bargain Network Inc. lets subscribers search its site for the lowest price on all sorts of merchandise, from cars and electronics to Pez dispensers. But how it gets those prices has put it on the front lines of an Internet legal war.

To provide subscribers with its custom-made bargain basement, Bargain Network deploys automated search robots -- known as bots, crawlers, spiders or scrapers -- to hundreds of online retailers and auctioneers. The bots collect the information and return it to be posted on the company's Web site. provides direct links to the retailer or auction site, so the best deal is just a click away for online shoppers.

But some Web sites object to the practice, arguing that such unauthorized harvesting of data is effectively trespassing on their sites. If consumers are automatically deep-linked directly to a specific piece of merchandise, they won't see ads and other services found on a site's home page, the sites argue. Also, lumping a bunch of vendors together for price-comparison purposes can dilute a business's most valuable asset: its brand identity.

Proponents of bots say that at the controversy's most basic level, placing limits on the ability of automatic robots to freely roam the Web is a threat to the fundamental freedom of the Internet. Such restrictions, they say, could be used to go after the very technologies that make the Web's sea of information manageable, such as search engines.

But for those battling the bots, the issue is equally simple: Companies that sell over the Internet argue that they have a right to protect their sites from businesses that mine their databases for information and then use that information for their own money-making purposes.

"There is a battle royal brewing around this issue," says Christian Hunter, founder and chief executive of Bargain Network, of Santa Barbara, Calif.

Bots are being used increasingly on the Web for a variety of business purposes, including mining data from competitors' sites, harvesting e-mail addresses for potential spam targets and offering quick links to aggregated retail and auction data for a fee, as Bargain Network does. Some search engines use bots, too. , for instance, sends bots to scour the Web and gather pages.

The difference is that Google's visits are typically welcomed by the sites it hits. That's because Google's bots are programmed to respect so-called robot-exclusion protocols, or electronic "no trespassing" signs, which some sites use to protect their contents. It's the bots that ignore these protocols, including 's bots, that have some Web site operators up in arms, and in some cases the sites are filing lawsuits.

To protect their turf, the sites being visited by bots are employing a legal doctrine dating back to the Middle Ages known as "trespass to chattels" and a law passed in the mid-1980s designed to prevent hackers from gaining access to government and corporate computer systems. So far, most courts have ruled against the companies sending their unwanted robots to Web sites if they were told to keep out, bowing to the arguments that sites have the right to stop unauthorized access.

"People are now understanding that while we often travel freely within the Internet, the Internet is definitely not free," says Jay Monahan, associate general counsel for intellectual property and litigation at online auction house eBay Inc. "Robots have to be programmed to respect the no-trespassing signs."

Online-auction and other Internet businesses say bots cause them a host of harms, from draining their system capacity to undermining business partnerships that depend on strong brand recognition and enticing consumers to spend time rummaging around their sites. Moreover, they say, bots steal business from them by misleading and confusing customers.

For the bot-using companies and their legal allies, however, restricting the use of search bots unfairly blocks competition by exerting monopolistic control over information that wouldn't ordinarily be subject to copyright protection. True, the bot business models rely on information gathered from other Web sites, but proponents argue this practice is legitimate because the information is already freely available to the public. Further, they contend that their bots are necessary for consumers to derive the full benefit of e-commerce. Bots help consumers, the proponents say, by allowing them to cast the widest net possible on the Web for merchandise, services and prices.

Some legal experts say the court rulings favoring Web sites with no-trespassing protocols go too far and threaten the core principles of the Internet: If information is available to any surfer with a browser, the experts say, it should be available to companies using automated search robots to harvest information, even for their own business purposes.

The law shouldn't say a Web retailer can put up a sign on its home page saying only people interested in buying products can visit, and competitors can't, "just as we don't allow the owner of a store in a mall to forbid competitors to walk in and notice their prices," says William Fisher III, a professor of intellectual property and Internet law at Harvard Law School and faculty director of the school's Berkman Center for Internet and Society. "Generally speaking, there are lots of social and economic benefits of free navigation on the Internet, and that includes aggregating information found on other sites," Mr. Fisher says. "Often there's not enough injury to the common good to justify forbidding activity of the search engine."

Last April, a real-estate Web site whose listings were picked up by bots from filed a lawsuit against Bargain Network, the site's parent. Alleging trespass to chattels -- defined today as any intentional interference with an owner's use or possession of property -- unfair competition and misappropriation, Inc. filed its suit in U.S. District Court in Los Angeles claiming that Bargain Network "misappropriated" and repackaged information from Homestore's Web site, , the official site of the National Association of Realtors. The suit sought an injunction barring the practice and any profits that Bargain Network had gained by using the information.

What was happening was that apartment- and house-hunters who logged onto were given a direct link to listings on , with detailed information, including how to contact the local broker or management company advertising the real estate. pays to display the so-called multiple listing services on its site. It makes its money by displaying sponsor links and advertisements and offering a host of related services and products. These ads and services were not seen by users accessing the listings through .

Homestore, based in Westlake Village, Calif., argued in its lawsuit that Bargain Network was interfering with its "contractual and prospective" relationships with existing and potential sponsors, advertisers and customers, and was depriving Homestore of the opportunity to collect demographic information from Internet users. Homestore said in its suit it didn't take kindly to Bargain Network's "free-riding" on its efforts and having its listings aggregated with those of competing real-estate Web sites.

In response to the suit, Bargain Network says that it believes its behavior was lawful. But in July, the company settled the case, in part because it had already decided to prune its real-estate offerings, says Mr. Hunter, the CEO. The company agreed to stop sending its bots to . But, saying it anticipates moving away from the "nonconsensual" real-estate market, Bargain Network still sends bots to real-estate sites with properties being offered directly by owners and continues to send bots to a host of other auction and retail sites.

Even before the Homestore suit, Bargain Network had tinkered with the way it uses its bots in an attempt to avoid legal problems, Mr. Hunter says. Instead of harvesting and storing large amounts of data -- a practice that has landed other search-bot users in legal trouble -- Bargain Network only dispatches bots after a customer has placed an inquiry, thus reducing the number of searches and putting less of a strain on the target system's capacity. Bargain Network argues this is less intrusive on other Web sites also because the searches are initiated by individual consumers, rather than by Bargain Network itself.

Homestore's lawyer, Jeffrey Weingart, a partner at Brown Raysman Millstein Felder & Steiner LLP in New York, was unimpressed with that line of reasoning. "We did the research and based on the facts as we understood them, we felt each and every claim [in the Homestore lawsuit] had a valid basis for those particular causes of action," he says.

In December 1999, eBay sued what it saw as a particularly egregious trespasser, Bidder's Edge Inc. The company, which subsequently went out of business, let consumers simultaneously search for merchandise on eBay and dozens of other auction sites. After comparing bids across many sites, would-be buyers could click to the actual auctioneer from a list compiled on the Bidder's Edge site.

According to eBay, bots used by Bidder's Edge captured 100,000 eBay Web pages a day for storage in its own database, ignoring eBay's electronic keep-out protocols. While eBay didn't claim the searches disrupted its site, it argued that Bidder's Edge was trespassing, taking up eBay system capacity and imposing an unstated cost on the auction site.

"This dispossession of part of our capacity is harm," says Mr. Monahan, the eBay attorney. Just as someone taking a car for a spin without permission "would be a harm even if the car doesn't come back with dents in it," he says.

Bidder's Edge, which was based in Burlington, Mass., said in response to the suit that trespass to chattels was "an antiquated, rarely used tort that does not apply," and that the company's activity "was nothing like the destructive and harmful activities of hacking and spamming."

But in mid-2000, a federal judge in San Jose, Calif., issued a preliminary injunction against Bidder's Edge, barring the company from using any automated query program to copy any part of eBay's auction database without authorization. Bidder's Edge closed its doors in February of the following year, and settled its suit with eBay soon after, paying the auction site an undisclosed sum.

Despite the settlement, former Bidder's Edge executives maintain their actions were proper. "It's kind of like you can go to a store and get circulars and do price shopping and no one gets too upset," says James Carney, the company's former chief executive. "It's no different than going into a supermarket and writing down prices for someone."

In another dispute, the bots in question were used to generate sales leads. Verio Inc., a Web-hosting unit of Nippon Telegraph & Telephone Corp., would trawl the online database of , a New York company that registers Internet domain names. Verio would then use the information to contact the companies and try to sell its services to them.

, after fielding angry calls from customers complaining that it sold their private data to telemarketers, filed suit against Verio in 2000 in New York federal court, accusing Verio of trespass to chattels and of violating the Computer Fraud and Abuse Act. The judge issued an injunction stopping Verio from sending its bots, but Verio has filed an appeal with the U.S. Second Circuit Court of Appeals in New York. A ruling is expected soon.

In its appeal, Verio, of Englewood, Colo., argues that is required to make public the names of those registering domain names -- known in the industry as "whois" information, after the search function for finding registrants' names. This public information, Verio contends, should be just that: publicly available.

"To Internet users, whois data is as accessible as any content published on a Times Square billboard," Verio said in its appeal. "In Times Square, no one imagines that a billboard owner can obtain viewers' assent to restrictive terms of use for information on the billboard simply by posting such terms on the same billboard."

countered that Verio was aware of the terms of use, which forbids use of its information for mass-marketing purposes, and knowingly and continuously violated those terms.

Jack Levy, 's general counsel, says the Internet regulators' original motivations in requiring registrars to make whois information freely available were worthwhile "public policy" considerations, such as helping government authorities track down fraudulent and criminal activity on the Internet. "That said, there are people who go ahead and abuse it," Mr. Levy says. Because it's publicly available doesn't mean "anyone can access it in any way they desire. That's not what was intended."

Legal experts and attorneys for Web companies are watching to see how appeals courts decide the question of cyber-trespass. For companies looking for a defense against unwanted search bots, favorable opinions would make it easier to sue over unwanted intrusions. "When companies believe they have been harmed by a cybertrespass," says Nels T. Lippert, an intellectual-property lawyer in New York, "they will have a higher confidence level and would therefore look to litigation."

Some, however, worry that if sites are able to sue without showing they suffered actual harm, this could encourage other lawsuits against general-use search bots, such as search engines. "Someone could tomorrow take out a suit against Google and say they've trespassed," says Danny Sullivan, editor of , a Web site devoted to the search-engine industry. Though Mr. Sullivan imagines Google would have a strong defense because it generally heeds no-trespassing signs, he believes the rulings give plaintiffs a legal foundation for such actions. After all, he says, even so-called Googlebots can "go nuts" sometimes.

Google has received a "minuscule" number of complaints, a company spokesman says, adding that Google offers a slew of information and tutorials to Webmasters on how to control the way Google bots search their sites. If the bots are hitting a small Web site over and over, it can cause that site to slow down. As for potential lawsuits, the Google spokesman says, "It's hard to speculate on the future."

Others in the industry fear that the courts may push search-engine businesses into what was once an unthinkable realm: shifting the burden to the searchers by requiring permission for robotic entry to Web sites.

Says Mr. Sullivan: "The big fear is that a court can come out with a ruling and say, `We decide that sites can't be spidered unless they have given permission.' Search engines couldn't go to every Web site and say, `Do you mind if we spider you?' If you required robots to ask permission before spidering, you would bring searching the Web to a halt."

---

Ms. Plitch is a staff reporter of Dow Jones Newswires in Jersey City, N.J. She can be reached at phyllis.plitch@

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download