Site Checker Stops When Robots.txt Contains Exclusions

This forum is designated to discuss AceBIT SEO Professionals Suite/Hello Engines! Professional.
Post Reply
richfulton
Newbie
Newbie
Posts: 1
Joined: December 1st, 2008, 11:23 pm

Site Checker Stops When Robots.txt Contains Exclusions

Post by richfulton » December 1st, 2008, 11:37 pm

On 6.58, the site checker (HTML Validator, Load Time Checker, Code Shrinker) appear to all stop when they first hit an excluded directory in the robots.txt file on my server. The remaining files on the server whether excluded or not are not checked.

What entry needs to be added to robots.txt to allow "Hello Engines! Professional" to spider the entire site?

Support (AceBIT)

Re: Site Checker Stops When Robots.txt Contains Exclusions

Post by Support (AceBIT) » December 2nd, 2008, 6:27 pm

Hello,

the Site Checker does not use the file robots.txt but only the embedded meta-tags "ROBOTS". If it contains "NOFOLLOW", the Site Checker does not follow links from the current page. Unfortunately this is not customizable in the current version.

Best regards

Post Reply