Re: cgi-binFrom Bob on 12 June '00
replying to cgi-bin posted by peter laurie
>Our site runs entirely from scripts that live in a cgi-bin directory.
>I understand that some spiders don't like to see 'cgi-bin' in a URL, so I've aliased this to 'bin'. Does anyone know if that would put them off?
Are you using robots.txt and crawler.html to tell the spiders where and where not to index? That may help. Search on those terms.