---

[ June Full List | Reply ]

---

Re: HELP! Need info on SE spiders.... From SimpleEnigma at Spider Hunter on 28 June '00 replying to Re: HELP! Need info on SE spiders.... posted by David A >>I just discovered a huge problem... I have a brand new server... and I configured it using a virtal name aliasing system that works very well. >> >>If I want to create a subdomain, I simply create a folder and request the page... no configuration needed. >> >>The only downfall is that it only works with browsers that use the HTTP/1.1 protocol, because it needs to know the full address that the browser is looking for. >> >>I did not think this would be a problem, since the last few versions of IE and Netscape support this protocol. >> >>I didn't even think about SE spiders!!!! >> >>I already know that Lycos is still using HTTP:/1.0, and thus cannot find any of my pages on my server. >> >>Does anyone know where I can find info on SE spiders and find out which one's, if any, are using the new HTTP:/1.1 protocol??? >> >>This is a HUGE problem... and my access and error logs are showing hit after hit resulting in pages not found... all from SE spiders. >> >>Any info or direction to other info sites would be appreciated. >> >>Thanks, >> >>Brett > >Everything and more > >http://www.spiderhunter.com/ I found a few hits coming from this forum and decided to see waht the topic was. Upon reading this, I started collecting data about what protocol the spider that visit my sites are using. Since I set this up I have seen 9 spiders: 6 from Inktomi, 1 INfoseek, 1 AltaVista, 2 misc. ALL of them have been using the HTTP/1.0 protocol. I will have a list of what protocol is used by each spider before next week. (i.e. I'll work on it this weekend.) The list will be updated as spiders visit any of my sites.

Spider HTTP protocol - Brett LaCrboix - 28/06/00

Reply to this message
Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:	(if you'd like to receive automatic replies by email)
Your Message
>>>I just discovered a huge problem... I have a brand new server... and I configured it using a virtal name aliasing system that works very well. >>> >>>If I want to create a subdomain, I simply create a folder and request the page... no configuration needed. >>> >>>The only downfall is that it only works with browsers that use the HTTP/1.1 protocol, because it needs to know the full address that the browser is looking for. >>> >>>I did not think this would be a problem, since the last few versions of IE and Netscape support this protocol. >>> >>>I didn't even think about SE spiders!!!! >>> >>>I already know that Lycos is still using HTTP:/1.0, and thus cannot find any of my pages on my server. >>> >>>Does anyone know where I can find info on SE spiders and find out which one's, if any, are using the new HTTP:/1.1 protocol??? >>> >>>This is a HUGE problem... and my access and error logs are showing hit after hit resulting in pages not found... all from SE spiders. >>> >>>Any info or direction to other info sites would be appreciated. >>> >>>Thanks, >>> >>>Brett >> >>Everything and more >> >>http://www.spiderhunter.com/ > > >I found a few hits coming from this forum and decided to see waht the topic was. Upon reading this, I started collecting data about what protocol the spider that visit my sites are using. Since I set this up I have seen 9 spiders: 6 from Inktomi, 1 INfoseek, 1 AltaVista, 2 misc. ALL of them have been using the HTTP/1.0 protocol. > >I will have a list of what protocol is used by each spider before next week. (i.e. I'll work on it this weekend.) The list will be updated as spiders visit any of my sites.