deadlock Message Board
[ October Full List | Reply ]

Re: OK to submit individual pages to Excite?
From Max on 30 October '98
adding to OK to submit individual pages to Excite? posted by Dave

>Instructions for submittal to Excite say to only submit the homepage.
>A newsletter I received recently says it's OK to submit individual
>pages.
So who do you believe? The people running the search engine or the
person writing the newsletter? Unless the newsletter comes FROM Excite,
I'd recommend following their rules.

>Right now my site has only been spidered down to pages one
>folder below my main directory and the number of those pages (as shown by Web
>Position Analyzer) is substantially less than the total number of pages
>that exist at that level. Does the spider only go one folder deep?
>Does it "skip around" or is it supposed to index all pages at the
>level it is working on?

Responsible search robots have a programmed limit of pages per unit of
time, to avoid swamping a site with requests. If you submit a large new
site, they will have to make several visits to get it all.
(Strategy tip: Get the main pages up and submitted as soon as you can, then
load the rest of the content in phases. They'll see the new links and dive
right for them.)
Some of them have a way of ranking the links to decide which to follow
next, others will take them in the order they occur on the page.
Searchenginewatch.com has some graphs and commentary on the observed
behavior of various robots on several sites that shows how erratic they
can be.



Replies from other people:


Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive
automatic replies by email)

Your Message