Moderators: Support (AceBIT), AceBIT
2 posts
‹ Page 1 of 1
Site Checker Stops When Robots.txt Contains Exclusions
by richfulton » Mon Dec 01, 2008 11:37 pm
On 6.58, the site checker (HTML Validator, Load Time Checker, Code Shrinker) appear to all stop when they first hit an excluded directory in the robots.txt file on my server. The remaining files on the server whether excluded or not are not checked.
What entry needs to be added to robots.txt to allow "Hello Engines! Professional" to spider the entire site?
What entry needs to be added to robots.txt to allow "Hello Engines! Professional" to spider the entire site?
- richfulton
- Rookie
- Posts: 1
- Joined: Mon Dec 01, 2008 11:23 pm
Re: Site Checker Stops When Robots.txt Contains Exclusions
by Support (AceBIT) » Tue Dec 02, 2008 6:27 pm
Hello,
the Site Checker does not use the file robots.txt but only the embedded meta-tags "ROBOTS". If it contains "NOFOLLOW", the Site Checker does not follow links from the current page. Unfortunately this is not customizable in the current version.
Best regards
the Site Checker does not use the file robots.txt but only the embedded meta-tags "ROBOTS". If it contains "NOFOLLOW", the Site Checker does not follow links from the current page. Unfortunately this is not customizable in the current version.
Best regards
- Support (AceBIT)
2 posts
‹ Page 1 of 1
Who is online
Registered users: Google [Bot], MSN [Bot], Yahoo [Bot]