---

[ March Full List | Reply ]

---

Re: robots.txt ERROR From David A on 29 March '00 replying to Re: robots.txt ERROR posted by T J Daniels The robots meta tag is not mandatory. You can use it if you want to give a spider specific commands for that page, like 'don't index this page but follow the links to other pages'. Both the robots meta tag and robots.txt file are optional. They have some overlapping functions and they have some unique functions. Heres a link to the robots.txt standard. http://info.webcrawler.com/mak/projects/robots/exclusion-admin.html >David, don't you have to have a robots.txt in your meta tags? I checked out Chucks page >and saw no 'robots.txt'. > > >>>Hi, >>> >>>In looking at my stats I noticed that under "Failed URL's" /st-croix/robots.txt was listed 11 times. Does this mean that the site was visited by robots 11 times and they couldn't get in? If so how do I fix it? Also, how do you tell when a robot has visited your site? >>> >>>Thanks, >>>Chuck >> >>robots.txt is a file that you can create to tell SE spiders which parts of your site to not index /images/cgi-bin/errors/etc. When a spider visits your site they will first look for this file (thus the 11 not found errors) then they will crawl your site in the manner that they are programed to.

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>The robots meta tag is not mandatory. You can use it if you want to give a spider specific commands for that page, like 'don't index this page but follow the links to other pages'. Both the robots meta tag and robots.txt file are optional. They have some overlapping functions and they have some unique functions. >Heres a link to the robots.txt standard. >http://info.webcrawler.com/mak/projects/robots/exclusion-admin.html > >>David, don't you have to have a robots.txt in your meta tags? I checked out Chucks page >>and saw no 'robots.txt'. >> >> >>>>Hi, >>>> >>>>In looking at my stats I noticed that under "Failed URL's" /st-croix/robots.txt was listed 11 times. Does this mean that the site was visited by robots 11 times and they couldn't get in? If so how do I fix it? Also, how do you tell when a robot has visited your site? >>>> >>>>Thanks, >>>>Chuck >>> >>>robots.txt is a file that you can create to tell SE spiders which parts of your site to not index /images/cgi-bin/errors/etc. When a spider visits your site they will first look for this file (thus the 11 not found errors) then they will crawl your site in the manner that they are programed to.