A
related article:
Internet
Condemned To Google HellAndy Greenberg 04.30.07, 6:00 AM ET
Don't anger the Google gods.
That's the lesson Paul Sanar learned--too late--last year. Up until last fall, the 21-year-old New Yorker depended solely on the search engine to keep traffic flowing to Skyfacet.com, his online diamond business; Sanar says he sold $3 million dollars worth of jewelry a year. Then, he says, Google (nasdaq: GOOG - news - people ) turned its back on Skyfacet.com, condemning the site to Internet obscurity.
Beginning in September 2006, Skyfacet no longer showed up on the first few pages of Google's results when users typed in search terms like "diamonds" and "engagement ring." The site's traffic vanished, and Sanar says his sales dropped $500,000 in three months.
What happened? Sanar isn't completely sure. But he does know that his site has been condemned to the supplemental index, a dreaded backwater region of Google search results that goes by another name in online marketing circles: Google Hell.
Google Hell is the worst fear of the untold numbers of companies that depend on search results to keep their business visible online. Getting stuck there means most users will never see the site, or at least many of the site's pages, when they enter certain keywords. And getting out can be next to impossible--because site operators often don't know what they did to get placed there.
Google's programmers appear to have created the supplemental index with the best intentions. It's designed to lighten the workload of Google's "spider," the algorithm that constantly combs and categorizes the Web's pages. Google uses the index as a holding pen for pages it deems to be of low quality or designed to appear artificially high in search results.
Those pages are scanned far less frequently than those in the main index, meaning that once a page is marked for Google Hell, it can languish there for as long as a year before Google even deigns it worthy of a reappraisal. And as Google tries to manage an explosively growing Web, more and more sites are finding themselves thrown into the search engine's digital dungeon.
If that makes the world's leading Web-crawler sound judgmental, consider Google's difficult position. The search juggernaut is faced with the endless task of reading and ranking the ever-expanding Web's billions of pages, the equivalent of putting the Earth's population in order from tallest to shortest every few minutes. Meanwhile there are growing numbers of pages filled only with junk text and advertising, designed solely to fool the engine. It's Google's task to sort out the trash from the worthwhile, and to do it better and faster than competitors like Yahoo! (nasdaq: YHOO - news - people ), Microsoft (nasdaq: MSFT - news - people ), or InterActiveCorp's (nasdaq: IACI - news - people ) Ask.com.
So how does Google decide what kind of pages get punished? That's where things get tricky. Google keeps the details of its decision-making a secret, since the company is trying to prevent sites from gaming the search engine. But it also means that site operators like Paul Sanar can offend Google and not know what they've done until its too late.
In retrospect, Sanar thinks he can trace his problem to a search marketing consultant he had paid $35,000 to improve Skyfacet's Google rankings. He now believes the consultant mistakenly replicated content on many of the site's pages, making them look like duplicate--that is, spam--content. But even after he reversed the consultant's changes, he couldn't get Skyfacet's pages out of Google Hell, where they remain today.
Other online businesses have similar stories. MySolitaire.com, another online diamond business, spent January to June of 2006 in the supplemental index. Amit Jhalani, the site's vice president of search marketing, says he figures that cost his business $250,000 in sales, and he says he still doesn't know why the site's pages got Google's thumbs-down.
"So many of the rules are vague," Jhalani says. But he admits that he tried gray-area tactics like buying links from more established sites to juice his traffic. "For a small site like ours, you have to stay right on the edge to compete with sites with bigger budgets," he confesses.
Jhalani says he removed the links that may have offended Google, but the site remained in Google's gulag. Jhalani wrote Google asking the search engine to reappraise MySolitaire; nothing happened. Since Google ranks sites partially by the quality of sites that link to them, he painstakingly contacted every site that seemed to be of low quality and linked to MySolitaire, asking them to remove their links, sometimes even sending cease-and-desist letters. Finally the site returned to Google's main index last June, though Jhalani has no way of knowing just what finally caused Google's algorithm to forgive him.
Chris Bartow is a search marketing consultant for Revenco.com, a real estate site that also saw the majority of its pages sent to Google Hell for six months of 2006. Bartow believes that some identical content on 90 of his site's property listing pages caused Google to mistake them for plagiarized spam sites. "I know they're trying to get rid of sites with no practical purpose," he says. " But when your pages get dumped, you lose half your traffic and a lot of money."
Bartow thinks his misfortune stemmed from a temporary glitch in Google's algorithm. But other search engine marketers say that Google Hell is only increasing in size and severity. "The supplemental index has been on the upswing for quite a while," says Aaron Wall, a search engine consultant and Google-watcher. "They've gotten much more aggressive about throwing pages in there."
Search marketer Michael Gray says he's seen the standards "tighten and loosen and tighten and loosen," but the last six months have been particularly brutal. "There has been a lot of collateral damage with some of these decisions," Gray says. He cites the growing sophistication of spam pages as one source of trouble. "Google's trying not to throw the baby out with the bathwater, but it's kind of impossible. A spammer can very easily create something that resembles a legitimate site if he knows the right tricks," he says.
The criteria for which pages are targeted for the supplemental index remains a subject of guesswork. But Web designers have found that pages with duplicate content, few words or pictures, and a lack of links to other quality sites are the most likely to be pulled in. Most agree that newly created sites are especially vulnerable.
As for Google's own take on its supplemental index, the company is typically tight-lipped. Google's official page for Webmasters cryptically notes that Google is "able to place fewer restraints on sites that we crawl for this supplemental index than we do on sites that are crawled for our main index," a phrase that puzzles most search marketers.
In an e-mail, Google product manager Prashanth Koppula offers little more in the way of an explanation. Asked if the supplemental index is getting bigger, he responds that "new pages are constantly being added," but that the "algorithmic nature" of Google's spider makes it hard to measure the index's size or how fast it's growing. That's not a problem, Koppula says, because supplemental results are no less legitimate than normal results, and pages in the supplemental index aren't checked any less frequently by Google's spider.
But Jim Boykin, another search marketing consultant and blogger, doesn't buy it. "If your page is in the supplementals, it won't rank for any competitive search, and it can be really hard to get it out," he says. "That's why we call it Google Hell."
