deadlock Message Board
[ March Full List | Reply ]

Re: robots.txt
From jwj on 20 March '01
replying to robots.txt posted by inachu

I have a site that I've been trying to get the robots to leave alone so I have been using the asterisk, as I assumed that was all you needed but slurp and gulliver keep showing up - I don't know if they're ignoring the request or simply coming to see if it's changed, so I now have the robots.txt file reading like this:

User-agent: *
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: Gulliver
Disallow: /

I'll check it next week and see if they're still sniffing.

As someone pointed out a few weeks ago, cnn is covering all angles on their file

Reply to this message

Required Details
Your name:

Optional Details
Home page title:
Home page URL:
(if you'd like to receive automatic replies by email)

Your Message