deadlock Message Board
[ October Full List | Reply ]

Re: Robots.txt did not work
From Jaime on 24 Oct '00
replying to Re: Robots.txt did not work posted by Tim

>>>Can anyone help me figuring out why my robots.txt file did not work? I submitted it to alltheweb SE and all of my pages had been indexed.
>>>http://www.gorgai.com/robots.txt
>>
>>
>>You might need to add the name of the spider for AllTheWeb - not sure exactly what it's called, but here are names for some of the other search engines.... Maybe try FAST - under the impression that they are connected with AllTheWeb in some way???!!!
>>
>>AltaVista - Scooter
>>Hotbot/AOL/Goto/MSN/Iwon/Looksmart - Slurp
>>Excite - Architext Spider
>>Google - Googlebot
>>Fast - Fast
>*******
>I don't think it should be problem since I use wildcard * for the User-agent. I might be wrong.

I had another thought - sometimes it depends on where you locate your txt file. Have you done any research by searching the web? I found some pretty good sites with lots of info on it.... Let me know how you get on!!!




Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message