>I'm looking for feedback from anyone who has
>used robots.txt files with *any* success.
I've been using it for a long time, mostly to keep the little critters
out of the experimental parts of some sites. The search engine robots
obey the exclusion protocol. The ones scavenging email addresses from
web sites usually don't follow the rules in robots.txt and end up caught
in WebPoison scripts where they recurse themselves to death.
It works the way I want it to. What were you trying to do with it?