Robots.txt - Standards

Hi friends,

Today I came across a very nice article about robots.txt on SEO Chat by James.

It covers topics of ranging how to create robots.txt for your web site, use of various syntax to allow and disallow various robots, search engine spiders to crawl your web site, internal directories and specific files, adding sitemaps auto-discovery, crawl-delay and adding comments in the robots.txt file.

Here is the link to the article.