Google has multiple "Danger Will Robinson" results

chris_at · Post by **chris_at** » Tue Nov 01, 2005 5:06 am

I googled for my website today, and I saw that that Google hat quite a few results for "Danger Will Robinson" pages. Apparently, Google tried to follow the trackback URLs.

I have been using 0.8.4 until I upgraded to 0.9 when it was released, and in 0.9, the trackback URLs have "nofollow" attributes which should theoretically prevent Google from doing this. I have no idea how it was in 0.8.4 though, could somebody give me more info on this, or how I could prevent it?

No urgency or anything, I'm just curious.

Thanks
Christian

chris_at · Post by **chris_at** » Tue Nov 01, 2005 6:22 pm

I just did a quick search for "type=trackback" in my web server log, and indeed, Googlebot is trying to crawl this URL.

Anything I can do to prevent this?

Post by **garvinhicking** » Thu Nov 03, 2005 9:32 am

Well, you should be able to alter your robots.txt for that cause - else Google *should* be paying attention to rel=nofollow, but this might take some days until it's reflected in their archive

Regards,
Garvin

chris_at · Post by **chris_at** » Thu Nov 03, 2005 8:32 pm

The robots.txt file can only exclude directories (which is weird. If a robot has enough juice to spider my entire site, it should be able to parse a simple regexp...)

I'll drop a line to Google feedback if the problem persists...

Thanks for the info though

Christian