Re: meta commands for robots|
From Hope on 14 June '01
replying to meta commands for robots posted by jwj
The command Index, Follow give an open invitation to the spiders to go through your entire site. Not all spiders follow this tag. You should use robots.txt for better control over a spiders move over your site.
>As DRB suggested,
>"<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
>The last line for Robots is an Important line to use."
>I do realize that the commands NOT to follow are important when they apply, but do you guys know whether there is any level of significance to the command to index and follow? Or might that be of no effect?