>How do you keep the search engines from visiting your pages?
You put a file into your site's root directory called "robots.txt"
that specifies where they are wanted. This MUST be a plain ASCII
text file, with the format shown here. the # are the comments.
It's easiest to keep them out of entire directories at a time
# robots.txt for http://www.yoursite.com/
User-agent: * # this means ALL of them, but you can be selective
Disallow: /thisdirectory/ # You tell them which directories are off
disallow: /thatdirectory/ # limits to them. Watch your log files
disallow: /anotherone/ # for requests for robots.txt and you can tell
# which robots have been by.
If you want to get any fancier than this, look up the
"Robot Exclusion Protocol" on Yahoo. there's a lot more that
can be done with it, but this is the easiest
WARNING: not all robots will obey this, especially the ones that are
scavenging email addresses. If the information is sensitive, keep
it on an unconnected hard drive or a password protected server.