deadlock Message Board
[ November Full List | Reply ]

Will the SEs still read my robots.txt?
From Nicholas on 6 Nov '99
I tossed a server-side include into my robots.txt, just to see who's
snooping. Works fine.

But bringing up the page in a browser, it shows:

User-agent: * Disallow: /cgi-bin Disallow: index.html Disallow: /mousetraps/ Disallow:
/mousetraps/index.html Disallow: /mousetraps/about.shtml

rather than

User-agent:*
Disallow: /cgi-bin
Disallow: index.html
Disallow: /mousetraps/
Disallow: /mousetraps/index.html
Disallow: /mousetraps/about.shtml

as it used to. I wonder if the spiders are still reading it correctly?
(It still looks OK in Doc Source view.)




Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message