Re: CGI and SEsFrom David A on 7 Dec '99
replying to Re: CGI and SEs posted by Nicholas
>>I've read in the help section of several SEs that URLs containing "cgi-bin" are not indexed. But what about using server side includes to include the cgi directly onto your page. I am in the process of testing this but thought I would ask if anybody had tried this?
>>I think the way its handled on the server end is that the server takes the request for the html doc finds any includes in it and sticks them in. Then sends the completed page to the user. It seems to me that the robots would not know this was a dynamically created page.
>>Heres the page I'm attempting to submit. Its mostly CGI generated but I don't think the SEs will be able to tell. What do you think?
>A spider won't see that page as dynamic; it's the "cgi" or "?"
>in the URLs that flag them. All the spider knows on an SSI page is
>that there is one blank line, which could have been blank for
>infinite innocuous reasons, and the page takes a split-second
>longer to deliver.
>But if you're planning to do "IP-specific" delivery, serving
>a different page to each spider, I'd be cautious; AV Scooter
>may not always arrive from an AV IP address.
I mentioned this because there was some discussion about people whose whole sites were cgi and product based. And, they were having a hard time getting those pages listed. This may be an alternative.