deadlock Message Board
[ August Full List | Reply ]

Re: stopping spidering
From Brian on 15 August '00
replying to stopping spidering posted by sammyh

Use the robots.txt on your server
Then you can ban any spider or software bot from users anytime you want.

Do a search for robots.txt
http://www.cnn.com/robots.txt a good example of what a robot.txt looks like.




Brian


>Somebody here must be able to help me - you all seem like very clever people (the old addage of flattery will get you everywhere!)
>
>What is the simplest and most effective way to stop a site being spidered? I only want my top level page indexed - as the rest of the site is frames based & I don't want everybody diving into random pages....
>
>OR, is there an HTML tag that I could put on my pages to prevent this - ie. re-directing to the "start" framespage if people come into a random page in the site (which should be viewed in frames)?
>
>thanks




Reply to this message

Required Details
Your name:
Subject:

Optional Details
Home page title:
Home page URL:
Email:
(if you'd like to receive automatic replies by email)

Your Message