---

[ April Full List | Reply ]

---

Re: What is crawler.html/crawlers.html? / How 2 use them?? From Dale on 21 April '01 replying to What is crawler.html/crawlers.html? / How 2 use them?? posted by WP1 >Hello, > >Where can I get some info on "crawler.html" and/or "crawlers.html". > >1. What exactly are they?? >2. Is it cool to use them?? (that is, do you get penalized by Search Engines for using them?) >3. Can it be used with "robots.txt" and "<META> tags?? > >Any and all information appreciated. > >WP Actually I new to this also, but it was suggested to me to use this as the last line of the Meta Tags. Not sure if its good or not, but I have been getting on search engines. <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> Maybe someone in here will read this and give some feedback on it.

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>>Hello, >> >>Where can I get some info on "crawler.html" and/or "crawlers.html". >> >>1. What exactly are they?? >>2. Is it cool to use them?? (that is, do you get penalized by Search Engines for using them?) >>3. Can it be used with "robots.txt" and "<META> tags?? >> >>Any and all information appreciated. >> >>WP > >Actually I new to this also, but it was suggested to me to use this as the last line of the Meta Tags. Not sure if its good or not, but I have been getting on search engines. > ><META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> > >Maybe someone in here will read this and give some feedback on it.