Robots.txt Analysis - Check whether your site can be accessed by Robots (Google,Yahoo,MSN)


With this free tool you will be able to test and analyze URL against robots.txt file. The tool will fetch the robots.txt file from the domain and it will parse (pattern matching supported) it to check whether the URL is allowed to be accessed by robots like googlebot, yahoo slurp, msnbot and others. The robots.txt file must be in the root of the domain. For example: for the url http://www.example.com/file/page.php the robots file need to be http://www.example.com/robots.txt. You can select the desired bot (user-agent) from below.
Note: You web pages still may not be indexed if they are blocked with meta tag by the robots that support it.

URL:

Robot: