This forum is designated to discuss AceBIT SEO Professionals Suite/Hello Engines! Professional.

Moderators: Support (AceBIT), AceBIT

Site Checker Stops When Robots.txt Contains Exclusions

Postby richfulton » Mon Dec 01, 2008 11:37 pm

On 6.58, the site checker (HTML Validator, Load Time Checker, Code Shrinker) appear to all stop when they first hit an excluded directory in the robots.txt file on my server. The remaining files on the server whether excluded or not are not checked.

What entry needs to be added to robots.txt to allow "Hello Engines! Professional" to spider the entire site?
richfulton
Rookie
Rookie
 
Posts: 1
Joined: Mon Dec 01, 2008 11:23 pm

Re: Site Checker Stops When Robots.txt Contains Exclusions

Postby Support (AceBIT) » Tue Dec 02, 2008 6:27 pm

Hello,

the Site Checker does not use the file robots.txt but only the embedded meta-tags "ROBOTS". If it contains "NOFOLLOW", the Site Checker does not follow links from the current page. Unfortunately this is not customizable in the current version.

Best regards
Support (AceBIT)
 

Return to AceBIT SEO Professionals Suite

Who is online

Registered users: Google [Bot], MSN [Bot], Yahoo [Bot]

Who is online

In total there are 5 users online :: 3 registered, 0 hidden and 2 guests (based on users active over the past 5 minutes)
Most users ever online was 100 on Sun Jul 15, 2007 10:38 pm

Registered users: Google [Bot], MSN [Bot], Yahoo [Bot]