---

[ March Full List | Reply ]

---

Re: robots.txt ERROR From T J Daniels at ''POETRY'' by: T. J. Daniels on 29 March '00 replying to Re: robots.txt ERROR posted by David A David, don't you have to have a robots.txt in your meta tags? I checked out Chucks page and saw no 'robots.txt'. >>Hi, >> >>In looking at my stats I noticed that under "Failed URL's" /st-croix/robots.txt was listed 11 times. Does this mean that the site was visited by robots 11 times and they couldn't get in? If so how do I fix it? Also, how do you tell when a robot has visited your site? >> >>Thanks, >>Chuck > >robots.txt is a file that you can create to tell SE spiders which parts of your site to not index /images/cgi-bin/errors/etc. When a spider visits your site they will first look for this file (thus the 11 not found errors) then they will crawl your site in the manner that they are programed to.

Re: robots.txt ERROR - David A - 29/03/00

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>David, don't you have to have a robots.txt in your meta tags? I checked out Chucks page >and saw no 'robots.txt'. > > >>>Hi, >>> >>>In looking at my stats I noticed that under "Failed URL's" /st-croix/robots.txt was listed 11 times. Does this mean that the site was visited by robots 11 times and they couldn't get in? If so how do I fix it? Also, how do you tell when a robot has visited your site? >>> >>>Thanks, >>>Chuck >> >>robots.txt is a file that you can create to tell SE spiders which parts of your site to not index /images/cgi-bin/errors/etc. When a spider visits your site they will first look for this file (thus the 11 not found errors) then they will crawl your site in the manner that they are programed to.