Seems that from your message states
is that part of your domain is already indexed
The only way part of it was index when in fact
all of it should have been is this.
ROBOTS.TXT put this in the root
and CHMOD it so webspiders can read it.
and in the robots.txt file
you can tell the spiders which ones it can index
Take a look at robots.txt file at
I dont think there is a copyright on robot.txt
files so I copied it also here:
Go check out roverbot.com!
>If you have a domain www.hardcore.com, say.
>AV will index max. 200 of your pages, right?
>But... what if www.hardcore.com happens to point to a subdirectory of another site that already has 200 pages indexed with AV?
>Are you screwed in this case? Does the 200 correspond only to your domain, or is the place its sitting important?