Recently I got a message from Google Webmaster saying my site’s robots.txt file is Unreachable .
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn’t crawl any pages listed in that file, we postponed our crawl. Your site’s overall robots.txt error rate is 50.0%.
But it exists and I can reach it in browser http://www.jeriffcheng.com/robots.txt
I tried Fetch as Google Tool in Webmaster and got the following answer:
It seems something is blocking Googlebot from accessing my site
- Or my domain? I thought it might be a domain access problem, maybe DNS resolve errors? I checked my DNS health http://www.intodns.com/jeriffcheng.com but found no errors. So, not Network problems.
- Then, I asked my neighbors on the same server and they had exactly the same problem. So, it might be a server-wide problem.
- I send email to the hoster , and got several replies, finally, I got the right answer:
Your hosting server was blocking Google from crawling. Our sys admins have removed it so please have Google Webmaster Tools retry and it should not be a problem.
When google is crawling the site, the google IP can get blocked temporarily due to the crawler making the server un-responsive. When this happens no one can access their site so there is a temporary block of the crawler. This typically does not happen but can if a user on the server is abusing the hosting server but we cannot resolve their issue with the crawler. At this time Google should be able to crawl your site without any problems.