deadlock Message Board
[ March Full List | Reply ]

Re: robots.txt
From Hope on 20 March '01
replying to robots.txt posted by inachu

If you want to stop all spiders from an area of your site, then the asterisk is what you are wanting. If you are concerned about a certain spider, then you need to name the spider.

>I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders.
>
>Does the asterisk work or must you specify every spider agent?
>
>
>
>
>Inachu




Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message