deadlock Message Board
[ May Full List | Reply ]

Robots.txt - Is it O.K.?
From Paul Keene at Israel Hotels on 19 May '00
Could someone please check my robots text for errors in particular the way I have named the robots:

User-Agent:Scooter/2.0 G.R.A.B. X2.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:Scooter/1.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:Arachnoidea
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:ArchitextSpider
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm

User-Agent:KIT-Fireball/2.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:BackRub/2.1
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:Slurp/2.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:magellan.htm

User-Agent:Lycos_Spider_(T-Rex)
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:Gulliver/1.2
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:alta.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:fido/1.0 Harvest/1.4.p12
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:Infoseek Sidewinder/0.9
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm

User-Agent:Mozilla/3.01 (Win95; I)
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm





Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message