deadlock Message Board
[ April Full List | Reply ]

Re: What is crawler.html/crawlers.html? / How 2 use them??
From Dale on 21 April '01
replying to What is crawler.html/crawlers.html? / How 2 use them?? posted by WP1

>Hello,
>
>Where can I get some info on "crawler.html" and/or "crawlers.html".
>
>1. What exactly are they??
>2. Is it cool to use them?? (that is, do you get penalized by Search Engines for using them?)
>3. Can it be used with "robots.txt" and "<META> tags??
>
>Any and all information appreciated.
>
>WP

Actually I new to this also, but it was suggested to me to use this as the last line of the Meta Tags. Not sure if its good or not, but I have been getting on search engines.

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

Maybe someone in here will read this and give some feedback on it.




Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message