[
May Full List
|
Reply
]
Robots.txt - Is it O.K.?
From
Paul Keene
at
Israel Hotels
on 19 May '00
Could someone please check my robots text for errors in particular the way I have named the robots:
User-Agent:Scooter/2.0 G.R.A.B. X2.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:Scooter/1.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:Arachnoidea
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:ArchitextSpider
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
User-Agent:KIT-Fireball/2.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:BackRub/2.1
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:Slurp/2.0
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:lycos.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:magellan.htm
User-Agent:Lycos_Spider_(T-Rex)
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:Gulliver/1.2
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:alta.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:fido/1.0 Harvest/1.4.p12
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:infosk.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:webcrawler.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:Infoseek Sidewinder/0.9
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
User-Agent:Mozilla/3.01 (Win95; I)
Disallow:booking_form.htm
Disallow:currency.htm
Disallow:received.htm
Disallow:resterms.htm
Disallow:centre.htm
Disallow:form_comments.htm
Disallow:form_confirmation.htm
Disallow:form_voucher.htm
Disallow:form_sent.htm
Disallow:alta.htm
Disallow:northa.htm
Disallow:goog.htm
Disallow:inkto.htm
Disallow:nlight.htm
Disallow:excite.htm
Disallow:aol.htm
Disallow:goto.htm
Disallow:msn.htm
Disallow:looksmt.htm
Disallow:magellan.htm
Reply to this message
Required Details
Your name:
Subject:
Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)
Your Message
>Could someone please check my robots text for errors in particular the way I have named the robots: > >User-Agent:Scooter/2.0 G.R.A.B. X2.0 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:Scooter/1.0 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:Arachnoidea >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:ArchitextSpider >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm > >User-Agent:KIT-Fireball/2.0 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:BackRub/2.1 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:Slurp/2.0 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:lycos.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:magellan.htm > >User-Agent:Lycos_Spider_(T-Rex) >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:Gulliver/1.2 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:alta.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:fido/1.0 Harvest/1.4.p12 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:infosk.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:webcrawler.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:Infoseek Sidewinder/0.9 >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm > >User-Agent:Mozilla/3.01 (Win95; I) >Disallow:booking_form.htm >Disallow:currency.htm >Disallow:received.htm >Disallow:resterms.htm >Disallow:centre.htm >Disallow:form_comments.htm >Disallow:form_confirmation.htm >Disallow:form_voucher.htm >Disallow:form_sent.htm >Disallow:alta.htm >Disallow:northa.htm >Disallow:goog.htm >Disallow:inkto.htm >Disallow:nlight.htm >Disallow:excite.htm >Disallow:aol.htm >Disallow:goto.htm >Disallow:msn.htm >Disallow:looksmt.htm >Disallow:magellan.htm
