Re: CGI and SEsFrom Nicholas on 6 Dec '99
replying to CGI and SEs posted by David A
>I've read in the help section of several SEs that URLs containing "cgi-bin" are not indexed. But what about using server side includes to include the cgi directly onto your page. I am in the process of testing this but thought I would ask if anybody had tried this?
>I think the way its handled on the server end is that the server takes the request for the html doc finds any includes in it and sticks them in. Then sends the completed page to the user. It seems to me that the robots would not know this was a dynamically created page.
>Heres the page I'm attempting to submit. Its mostly CGI generated but I don't think the SEs will be able to tell. What do you think?
A spider won't see that page as dynamic; it's the "cgi" or "?"
in the URLs that flag them. All the spider knows on an SSI page is
that there is one blank line, which could have been blank for
infinite innocuous reasons, and the page takes a split-second
longer to deliver.
But if you're planning to do "IP-specific" delivery, serving
a different page to each spider, I'd be cautious; AV Scooter
may not always arrive from an AV IP address.