---

[ October Full List | Reply ]

---

Re: Robots.txt did not work From Jaime on 23 Oct '00 replying to Robots.txt did not work posted by Tim >Can anyone help me figuring out why my robots.txt file did not work? I submitted it to alltheweb SE and all of my pages had been indexed. >http://www.gorgai.com/robots.txt You might need to add the name of the spider for AllTheWeb - not sure exactly what it's called, but here are names for some of the other search engines.... Maybe try FAST - under the impression that they are connected with AllTheWeb in some way???!!! AltaVista - Scooter Hotbot/AOL/Goto/MSN/Iwon/Looksmart - Slurp Excite - Architext Spider Google - Googlebot Fast - Fast

Re: Robots.txt did not work - Tim - 23/10/00 Re: Robots.txt did not work - Jaime - 24/10/00 Re: Robots.txt did not work - Tim - 24/10/00 Re: Robots.txt did not work - Jaime - 26/10/00

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>>Can anyone help me figuring out why my robots.txt file did not work? I submitted it to alltheweb SE and all of my pages had been indexed. >>http://www.gorgai.com/robots.txt > > >You might need to add the name of the spider for AllTheWeb - not sure exactly what it's called, but here are names for some of the other search engines.... Maybe try FAST - under the impression that they are connected with AllTheWeb in some way???!!! > >AltaVista - Scooter >Hotbot/AOL/Goto/MSN/Iwon/Looksmart - Slurp >Excite - Architext Spider >Google - Googlebot >Fast - Fast