---

[ March Full List | Reply ]

---

Re: robots.txt From Hope on 20 March '01 replying to robots.txt posted by inachu If you want to stop all spiders from an area of your site, then the asterisk is what you are wanting. If you are concerned about a certain spider, then you need to name the spider. >I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders. > >Does the asterisk work or must you specify every spider agent? > > > > >Inachu

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>If you want to stop all spiders from an area of your site, then the asterisk is what you are wanting. If you are concerned about a certain spider, then you need to name the spider. > >>I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders. >> >>Does the asterisk work or must you specify every spider agent? >> >> >> >> >>Inachu