---

[ June Full List | Reply ]

---

Spider HTTP protocol From Brett LaCrboix at Sources4You.net on 28 June '00 replying to Re: HELP! Need info on SE spiders.... posted by SimpleEnigma >I found a few hits coming from this forum and decided to see waht the topic was. Upon reading this, I started collecting data about what protocol the spider that visit my sites are using. Since I set this up I have seen 9 spiders: 6 from Inktomi, 1 INfoseek, 1 AltaVista, 2 misc. ALL of them have been using the HTTP/1.0 protocol. > >I will have a list of what protocol is used by each spider before next week. (i.e. I'll work on it this weekend.) The list will be updated as spiders visit any of my sites. This will be interesting to see... I have already begun to get indexed... at least on Inktomi, which is actually what I wanted. I am not sure if Inktomi has some spiders running HTTP/1.0 and some running HTTP/1.1 Unless my server receives the full address from the spider (as in the HTTP/1.1 protocol) the result will be a 404 error. Lycos has a bunch of spiders... all HTTP/1.0 ... so I do not submit to them. If I wasn't starting to to get indexed by "some" search engines, then looking at my access logs would be very discouraging. 404 errors all over and all from spiders. On another note... Lycos does seem to be quick at sending out a spider after your submission... just too bad that does me no good. I will be watching http://www.spiderhunter.com for the results. -Brett

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
> >>I found a few hits coming from this forum and decided to see waht the topic was. Upon reading this, I started collecting data about what protocol the spider that visit my sites are using. Since I set this up I have seen 9 spiders: 6 from Inktomi, 1 INfoseek, 1 AltaVista, 2 misc. ALL of them have been using the HTTP/1.0 protocol. >> >>I will have a list of what protocol is used by each spider before next week. (i.e. I'll work on it this weekend.) The list will be updated as spiders visit any of my sites. > > >This will be interesting to see... I have already begun to get indexed... at least on Inktomi, which is actually what I wanted. > >I am not sure if Inktomi has some spiders running HTTP/1.0 and some running HTTP/1.1 > >Unless my server receives the full address from the spider (as in the HTTP/1.1 protocol) the result will be a 404 error. > >Lycos has a bunch of spiders... all HTTP/1.0 ... so I do not submit to them. > >If I wasn't starting to to get indexed by "some" search engines, then looking at my access logs would be very discouraging. 404 errors all over and all from spiders. > >On another note... Lycos does seem to be quick at sending out a spider after your submission... just too bad that does me no good. > >I will be watching http://www.spiderhunter.com for the results. > >-Brett