The Purpose Of "robots.txt" For A Website | Blue Eyed Paul I found the post to be a useful introduction to what a robots.txt file is and isn't. I'd read some forum posts etc about the "SEO benefits" so it's good to have that one cleared up. What Is "robots.txt"? It is a file placed in the root directory of a website that provides some instructions for a crawler/spider (Bot) about which files and/or directories are not to be indexed. It can be used to instruct a specific Bot or all Bots. internet in Public bookmarkswith googlerobots.txtseowebmaster