Jim's Message Board
[ Reply To This Message ]

Re: Keeping Pages Private
From Max at My Place - Finally! on 01 September '98
adding to Keeping Pages Private posted by Sam Hanie

>How do you keep the search engines from visiting your pages?

You put a file into your site's root directory called "robots.txt"
that specifies where they are wanted. This MUST be a plain ASCII
text file, with the format shown here. the # are the comments.
It's easiest to keep them out of entire directories at a time

# robots.txt for http://www.yoursite.com/

User-agent: * # this means ALL of them, but you can be selective
Disallow: /thisdirectory/ # You tell them which directories are off
disallow: /thatdirectory/ # limits to them. Watch your log files
disallow: /anotherone/ # for requests for robots.txt and you can tell
# which robots have been by.

If you want to get any fancier than this, look up the
"Robot Exclusion Protocol" on Yahoo. there's a lot more that
can be done with it, but this is the easiest

WARNING: not all robots will obey this, especially the ones that are
scavenging email addresses. If the information is sensitive, keep
it on an unconnected hard drive or a password protected server.

Replies from other people:

Reply to this message:

Who Are You?
Your name:
Home page title:
Home page URL:

If you'd like to get replies to your message by email, enter your address:

Your Reply: