Re: robots.txt ERRORFrom David A on 29 March '00
replying to robots.txt ERROR posted by chuck
>In looking at my stats I noticed that under "Failed URL's" /st-croix/robots.txt was listed 11 times. Does this mean that the site was visited by robots 11 times and they couldn't get in? If so how do I fix it? Also, how do you tell when a robot has visited your site?
robots.txt is a file that you can create to tell SE spiders which parts of your site to not index /images/cgi-bin/errors/etc. When a spider visits your site they will first look for this file (thus the 11 not found errors) then they will crawl your site in the manner that they are programed to.