robots.txt

[ March Full List | Reply ]

robots.txt

From inachu on 20 March '01

I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders.

Does the asterisk work or must you specify every spider agent?

Inachu

Re: robots.txt - Hope - 20/03/01
Re: robots.txt - jwj - 20/03/01
- Re: robots.txt - Inachu - 20/03/01
Re: robots.txt - David A - 20/03/01
- Re: robots.txt - Inachu - 20/03/01

*Reply to this message*
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders. > >Does the asterisk work or must you specify every spider agent? > > > > >Inachu