Why can't I get OUT of Yahoo! search?

There are so many sites clamouring to be on search engine site results pages, why waste your time on sites that don't want to be there?

I have many domains. Some, like this one, have lots of content. Some I'm building the content up. And some have absolutely nothing up yet.

Of all of those, some I just don't want on search engines. This one in particular. If you know who I am, you can find me. But I'm not interested in random people finding me. I post all the pages I want random people to read over on my dot-com site. Sure, I could password protect this site, but then my Mom, my ex-coworkers, or the casual friend (or even the boyfriend from high school whom I do want to contact me) can't just pop in and see what's going on.

To request search engines spidering a site (i.e. loading every page and following every link on the page) not search a particular page or part of a site, a site own needs to set up a robots.txt file. This file says which search engines can view what part of the site.

Only it just requests the search engine spiders to limit themselves to particular parts of the site. It doesn't actually stop them from viewing the site.

My robots.txt file basically says, "Don't search this site. Go away." I'm not interested in having search engines hit this site. This site is for me and having the search engines crawl this site amounts to stealing money from me in terms of bandwidth and processor time. I don't like it.

Most search engines honor the robots.txt file religiously. Google is great about it (see?). Some are less good (MSN). And some (Yahoo!) completely ignore the robots.txt file while claiming they honor it.

I sent an email in April asking them to remove my site. They replied with a form letter telling me to fix my robots.txt file. When I responded my robots.txt was exactly to the specification they sent to me, the search results dropped from 900+ down to about 500. A month later, they're back up to over 1050.

I sent another email to them today. This time, much to my chagrin, I threatened legal action. I hate doing so. I hate even suggesting contribution to the litigious mentality that seems to permeate modern culture (They look at you wrong? Sue them! Not doing what you want? Sue them! They type your name wrong? Sue them!). But I'm not sure what else I can do to get them to honor my requests.

Boo.

• Log in to read full content