Re: robots.txt

[ March Full List | Reply ]

Re: robots.txt

From David A on 20 March '01
replying to robots.txt posted by inachu

>I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders.
>
>Does the asterisk work or must you specify every spider agent?
>
>
>
>
>Inachu

Typically the asterisk is a wildcard meaning "all".

Re: robots.txt - Inachu - 20/03/01

*Reply to this message*
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>>I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders. >> >>Does the asterisk work or must you specify every spider agent? >> >> >> >> >>Inachu > >Typically the asterisk is a wildcard meaning "all".