Re: robots.txtFrom Inachu on 20 March '01
replying to Re: robots.txt posted by David A
Yes I agree but <img src="*.*> would never work but I wish the web was flexible like this....
>>I noticed palm(http://www.palm.com/robots.txt) has their robots.txt file just uses an asterisk to ban spiders.
>>Does the asterisk work or must you specify every spider agent?
>Typically the asterisk is a wildcard meaning "all".