http://netvouz.com/url/0a1bcd068779bac9501b72d81dfc016b?feed=rss Info about a link http://www.nextthing.org/archives/2007/03/12/robotstxt-adventure Last October I got bored and set my spider loose on the robots.txt files of the world. Having had a good deal of positive feedback on my HTTP Headers survey, I had decided to poke around in robots.txt files and see what sorts of interesting things I could find. fjordaan Thu, 04 Oct 2007 12:06:04 GMT