Robots.txt for SEOs – The Definitive Guide..
From Anne Smarty at Search Engine Journal
While nowadays almost every webmaster knows its basics, some things still cause misunderstandings
Robots.txt can prevent bots from crawling the page or directory but not from indexing or ranking the URL when it’s found via external references. In this case search engines will use information from these external sources to make judgments about the page and also to formulate the snippet (title and description that appear in search results
If you have both general (i.e. wildcard *) and specific (e.g. User-agent: googlebot ) user-agent sections, keep in mind that Google (and other crawlers) will only follow the most specific section and ignore all other sections (including the general one): thus repeat all the directives from general section in all specific ones.
The matching is from the left to the right, meaning that crawlers are blocked from anything that begins with / pattern. So if you have blocked yoursite.com/a directory, for example, keep in mind that you are also blocking all directories/ pages going after the root and starting with ‘a’ (e.g. yoursite.com/about). The related case was described in the recent WebmasterWorld thread.
To be on the safe
