Re: What is crawler.html/crawlers.html? / How 2 use them??|
From Dale on 21 April '01
replying to What is crawler.html/crawlers.html? / How 2 use them?? posted by WP1
>Where can I get some info on "crawler.html" and/or "crawlers.html".
>1. What exactly are they??
>2. Is it cool to use them?? (that is, do you get penalized by Search Engines for using them?)
>3. Can it be used with "robots.txt" and "<META> tags??
>Any and all information appreciated.
Actually I new to this also, but it was suggested to me to use this as the last line of the Meta Tags. Not sure if its good or not, but I have been getting on search engines.
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
Maybe someone in here will read this and give some feedback on it.