Great board! Appreciate the helpful spirit here. I have a dilema and wonder if anyone has gone through this and if so what's a exhausted webmaster to do? I have been reading how some of the search engines (Excite) will drop you if you submit too many pages. I have been very careful not to do that, but what if your competitor is submitting them for you to get your booted! I feel this is happening because I frequently see references to the submissions on our server logs on days when I have not done it.
I could of course return the favor and over submit theirs, but is there a better solution. I thought about writing Excite about the issue, but we are just one site of millions, why would they care... of course, they wouldn't! Besides, they would believe it wasn't us that was over submitting.
Any ideas on how to combat this???
Also has anyone ever seen a reference to a site called seventwentyfour.com in their server logs. It is a service that you can pay for to monitor your URL for broken links, etc. We have not requested this service. Their robot is called LinkWalker. I know there is a LinkWalker browser too. Are these different animals? I have written a robots.txt to disallow the LinkWalker robot from our site (was coming daily) but will this cause any problems getting listed anywhere else. I was just afraid their robot might perform indexing services elsewhere that I am not aware of.
Sorry for the long post... Any advice would be appreciated.