• The Purpose Of "robots.txt" For A Website | Blue Eyed Paul
  I found the post to be a useful introduction to what a robots.txt file is and isn't. I'd read some forum posts etc about the "SEO benefits" so it's good to have that one cleared up.
  What Is "robots.txt"? It is a file placed in the root directory of a website that provides some instructions for a crawler/spider (Bot) about which files and/or directories are not to be indexed. It can be used to instruct a specific Bot or all Bots.
  internet
  in Public bookmarks with google robots.txt seo webmaster

robots.txt from all users