[ Solved ] Webmaster Tools says Unreachable robots.txt, but it exists

Recently I got a message from Google Webmaster saying my site’s robots.txt file is Unreachable .

Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn’t crawl any pages listed in that file, we postponed our crawl. Your site’s overall robots.txt error rate is 50.0%.

But it exists and I can reach it in browser http://www.jeriffcheng.com/robots.txt

Google Bot

Google Bot


I tried Fetch as Google Tool in Webmaster and got the following answer:

Unreachable robots.txt

It seems something is blocking Googlebot from accessing my site

  • Or my domain? I thought it might be a domain access problem, maybe DNS resolve errors?   I checked my DNS health http://www.intodns.com/jeriffcheng.com  but found no errors. So, not Network problems.
  • Then, I asked my neighbors on the same server and they had exactly the same problem. So, it might be a server-wide problem.
  • I send email to the hoster , and got several replies, finally, I got the right answer:

Your hosting server was blocking Google from crawling. Our sys admins have removed it so please have Google Webmaster Tools retry and it should not be a problem.

When google is crawling the site, the google IP can get blocked temporarily due to the crawler making the server un-responsive. When this happens no one can access their site so there is a temporary block of the crawler. This typically does not happen but can if a user on the server is abusing the hosting server but we cannot resolve their issue with the crawler. At this time Google should be able to crawl your site without any problems.

  • Now my problem is solved, Google Bot is allowed to crawl my site/server. If you ever meet this problem, just send email to your hoster (Godaddy, Bluehost, Hostgator etc.) and ask them if Googlebot was blocked by the server.reachable robots.txt

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>